WARC archives

Capture pages to WARC, extract from archives at high volume, and replay them.

webfold capture <url> -o out.warc.gz [--backend native] [--tlp RED|AMBER|AMBER+STRICT|GREEN] [--retention-days N] [--no-gzip]
webfold warc   <file.warc|file.warc.gz> [--threads n] [--max n]
webfold dump   <warc-file> <out-file> [--max n]
webfold replay <file.warc|file.warc.gz> [--out replay.html] [--max n] [--full-html]

Command	Does
`capture`	Fetch one URL through the native backend into a WARC and a `.cdx` index (warcinfo, request, response and metadata records).
`warc`	Run extraction over a WARC or WARC.GZ stream and print a summary. Use `--threads` to parallelise a raw `.warc`; a `.warc.gz` is streamed on one thread and ignores `--threads`.
`dump`	Write the response pages out of a WARC so another tool can run over the identical corpus.
`replay`	Join each record’s raw HTML with its canonical extraction into a single-file HTML audit artefact.

Throughput depends heavily on hardware, thread count and corpus. See Benchmarks for what has been measured and under which protocol.