Format · WARC
Capture pages to WARC, extract text from archives and replay records as one HTML file.
Webfold captures pages into WARC 1.0 or 1.1 files, one gzip member per record so a single record can be read from its offset and length, extracts text from archived responses and replays records as one HTML file; conformance to the WARC standard is unverified because no test runs a stock WARC reader, and replay runs no JavaScript.
Webfold writes WARC files holding a warcinfo, request, response and metadata record per capture, with CDX and per-record index sidecars. It reads WARC/1.0 and WARC/1.1, plain or gzipped, and each captured record is its own gzip member, so one record can be decoded from its recorded offset and length. The archive commands extract text from response records and write one HTML replay file.
See it work
How do I try it from wherever I work?
1webfold capture https://example.com -o example.warc.gz --tlp AMBER --retention-days 365
Input
What input does WARC reading accept?
WARC/1.0 and WARC/1.1
The reader accepts exactly the version lines WARC/1.0 and WARC/1.1. WARC/1.1 with an application/http block is the default write shape. WARC/1.0 body-only records are the shape for engine-rendered DOM dumps, and the reader accepts both.
Plain and gzip files
A file is treated as gzip when it starts with the gzip magic bytes. read_all_path decodes every member in order. read_record_at decodes one member from a byte range and rejects a range that holds zero or more than one record.
Any WARC-Type
The reader does not filter record types, and a missing WARC-Type reads as an empty type. replay keeps response, resource and empty types and skips the rest.
HTTP messages inside records
A block whose Content-Type is application/http is parsed for its status or request line and its headers, with folded continuation lines joined. A legacy record whose Content-Type is the page's own MIME type is read as body only.
Capture input
`webfold capture` takes an http or https URL with the native backend only. Tor is refused in this binary. The MCP capture tool refuses non-http URLs such as file://.
Output
What comes out?
Four-record captures
Each capture writes warcinfo, request, response and metadata records in that order. The response carries the HTTP message in the default WARC/1.1 shape, and the request record carries the request headers with credentials redacted by default.
CDX and index sidecars
The .cdx file has one line per response record with 8 columns: SURT, timestamp, URL, MIME, status, digest, offset and length. The .idx file has one line per record with type, record ID, offset, length, concurrent-to IDs and two digests, with - for an empty field.
Digests
WARC-Block-Digest and WARC-Payload-Digest are written as sha256 with base32 values. The block digest covers different bytes in WARC/1.0 and WARC/1.1, and the reader can check either one when asked.
Replay HTML
One HTML file with a navigation index, per-record metadata, the webfold Markdown with its count of content blocks, and the raw HTML in a collapsible block truncated to 50 KiB unless --full-html is given.
Extraction summary
`webfold warc` prints the page count, pages per second and megabytes of HTML per second. It does not write Markdown to disk.
Citation identity
The MCP capture tool reports url, final_url, status, fetched_at, warc_file, record_id, offset, length, payload_digest and content_sha, and it picks the output path itself inside the archive directory.
How it works
What does the code do, step by step?
- 1
Fetch
`webfold capture` fetches the URL with the native fetcher and writes the response it receives. It does not re-render the page.
- 2
Write one member per record
WarcWriter::new_gz writes each record as one gzip member, in the order warcinfo, request, response, metadata. Request headers Authorization, Proxy-Authorization, Cookie and Cookie2 are replaced with [REDACTED] by default, and Set-Cookie is kept.
- 3
Normalise transfer coding at write
A body that the transport already de-chunked loses its Transfer-Encoding header and gets a Content-Length for the stored bytes. A Content-Encoding body is stored as received, and its payload digest covers those bytes.
- 4
Record placement in sidecars
Each record's file offset and length are recorded. The .cdx line of a response carries the true offset and length, and the .idx line lists every record with its record ID and digests.
- 5
Read
WarcReaderV2 reads the version line, splits each header at a colon and space, reads exactly Content-Length bytes, then splits the HTTP head off application/http blocks. It does not check digests or decode codings.
- 6
Extract or replay
`webfold warc` takes the payload of each response, runs to_blocks and render, and counts the output. `webfold replay` reads all records, decodes gzip or deflate content coding, runs the same converter and writes one HTML file.
Evidence
How do we know each claim?
Each row says how we know. Read from the code means the behaviour is in the implementation at the path shown. Covered by a test names a test in the repository; this page does not claim it was run for this release. Measured points to a benchmark with its protocol and limits.
| Claim | How we know | Where | Detail |
|---|---|---|---|
| Each capture writes four record types: warcinfo, request, response and metadata. | Read from the code | webfold/src/warc/mod.rs:17-28 | The module doc table lists the four WARC-Type values and their roles. roundtrip_tests.rs:90 write_then_read_all_returns_the_four_record_set_with_matching_ids asserts the order (not run). |
| Each record is one self-contained gzip member, so a record can be decoded from its offset and length. | Read from the code | webfold/src/warc/mod.rs:80-86; webfold/src/warc/random.rs:21-60 | read_record_at reads exactly length bytes at offset and decodes that range as one member. http_tests.rs:300 gzip_output_is_one_member_per_record checks the framing (not run). |
| The reader accepts only the WARC/1.0 and WARC/1.1 version lines. | Read from the code | webfold/src/warc/reader.rs:188-198 | Any other version line returns WarcReadError::Malformed with the message expected WARC/1.0 or WARC/1.1 version line. |
| The reader does not decode Transfer-Encoding or Content-Encoding. | Read from the code | webfold/src/warc/reader.rs:44-46; webfold/src/warc/http.rs:31-42 | The ReadRecord doc says the reader never decodes Transfer-Encoding or Content-Encoding. http.rs documents that chunked bodies are de-chunked by the transport before writing, and that other transfer codings are not decoded. |
| The reader does not verify digests while reading. Verification is a separate call. | Read from the code | webfold/src/warc/reader.rs:174-295; webfold/src/warc/mod.rs:126-148 | next_record does not call a digest check. verify_block_digest, verify_block_digest_warc10 and verify_payload_digest are public functions that the caller must invoke. |
| Headers are split at colon and space, and the body buffer is sized from Content-Length before it is read. | Read from the code | webfold/src/warc/reader.rs:216; webfold/src/warc/reader.rs:239-247 | A line without ': ' returns Malformed. body.resize(len, 0) runs before read_exact, so the allocation follows the claimed length. |
| The CDX has 8 columns with true offset and length; the index has 7 columns. | Read from the code | webfold/src/warc/mod.rs:743-765; webfold/src/warc/index.rs:92-112 | cdx_line writes surt, timestamp, url, mime, status, digest, offset, length. index_line writes warc_type, record_id, offset, length, concurrent-to, block digest and payload digest, using - for empty fields. |
| The threads option applies only to raw .warc files. A .warc.gz runs single-threaded. | Read from the code | webfold/src/main.rs:748-759; webfold/src/main.rs:755 | The warc branch calls bench_warc_par only when the path ends with .warc, and bench_warc (no thread argument) otherwise. |
| The raw .warc path reads the whole file into memory, and .warc.gz is decompressed by an external gunzip process. | Read from the code | webfold/src/cli/warc.rs:9-18; webfold/src/cli/warc.rs:220-223 | bench_warc spawns gunzip -c and expects it to be on PATH. bench_warc_par calls std::fs::read on the whole file. ARCHITECTURE.md:145-148 says the same for .warc.gz. |
| Round-trip, corruption and offset-agreement tests exist for both record shapes. | Covered by a test | webfold/src/warc/roundtrip_tests.rs:90, 211, 270, 312, 422, 531 | Named tests include write_then_read_all_returns_the_four_record_set_with_matching_ids, corrupted_digest_is_detected, cdx_and_index_agree_on_the_true_record_offset, every_record_offset_is_distinct_and_non_overlapping and http_block_corrupted_digest_is_detected. Read, not run. REPORT-warc-mcp-index.md (Verification) records a 40-test warc run from the author's worktree; that run was not repeated here. |
| HTTP chunking, content-coding and credential redaction have named tests. | Covered by a test | webfold/src/warc/http_tests.rs:300, 424, 456, 516, 541 | gzip_output_is_one_member_per_record, chunked_transfer_encoding_is_replaced_by_content_length, content_encoding_gzip_body_is_stored_as_received, default_redacts_request_credentials_but_not_set_cookie and opt_in_response_redaction_and_opt_out_request_redaction. Read, not run. |
| Random access: seeking to a recorded offset decodes exactly one member, and stale offsets and corrupted payloads are caught. | Covered by a test | webfold/src/warc/random.rs:106, 154, 194 | seeks_to_a_recorded_offset_and_decodes_exactly_one_member, stale_offset_is_rejected and corrupted_payload_breaks_digest_verification. Read, not run. |
| Throughput by configuration on an 84,749-page Common Crawl subset (15.9 GB). | Measured | benchmark slug warc-throughput; webfold/examples/seedbench.rs:20-30 | The slug records 26,675 pages/s for the seed path at 8 threads (4.8 GB peak), 14,238 pages/s for raw WARC at 8 threads (15.2 GB peak) and 2,943 pages/s for the seed path at 1 thread (35 MB peak). seedbench.rs attributes the two 8-thread figures to a seed paper and states that none is reproduced in that file. These are cited, not re-run. |
| Replay truncates raw HTML, decodes gzip and deflate, and reports a chrome-dropped count that is always zero. | Read from the code | webfold/src/cli/replay.rs:20-24; webfold/src/cli/replay.rs:236-262; webfold/src/cli/replay.rs:370; webfold/src/html_extract.rs:1149-1162 | RAW_HTML_LIMIT is 50 KiB. decoded_payload handles gzip, x-gzip and deflate with a 64 MiB cap and leaves other codings as stored. replay calls render(&blocks, None, 1), and render increments the dropped count only when a template is passed, so the count is always 0. |
What are the limits?
Know the limits
- Standards: no test in the tree runs a stock WARC reader. Comments in webfold/src/warc/records.rs name warcio, cdx_toolkit and pywb, but nothing runs them. The digests are labelled sha256 where the same file's comment says the WARC 1.1 convention is sha1 (webfold/src/warc/records.rs:44-50). ISO 28500 is not cited in the code, and we did not read the ISO text, which is a paid standard. Treat standards compliance as unverified.
- Memory: read_all and read_all_path return every record in one Vec, and the body buffer is sized from Content-Length before it is read, so a large claimed length allocates up front (webfold/src/warc/reader.rs:239-247; webfold/src/warc/reader.rs:320-330). read_record_at is bounded by the length it is given.
- Strictness: a header line without ': ' is rejected, while a record without WARC-Type is accepted with an empty type (webfold/src/warc/reader.rs:216; webfold/src/warc/reader.rs:259-264).
- Codings: the reader does not decode Content-Encoding, and the webfold warc harness does not either, so a gzip body is read as bytes there (webfold/src/warc/http.rs:40-42; webfold/src/cli/warc.rs). Replay decodes gzip and deflate only; br and zstd are kept as stored with a note (webfold/src/cli/replay.rs:236-262).
- Lookup: no command searches an archive by URL. The index is read by library functions and by the MCP capture tool, and the citation triple (record_id, offset, length) is the lookup key (webfold/src/warc/index.rs:50; webfold/src/warc/random.rs:69-76).
- Digests are checked only when the caller asks. The reader returns a corrupted record as it is unless verify_block_digest or verify_payload_digest is called (webfold/src/warc/reader.rs:174-295; webfold/src/warc/mod.rs:126-148).
- Throughput: the figures are cited from a benchmark record and a paper, and this repository does not reproduce them (examples/seedbench.rs:20-30). The warc command counts pages and bytes and writes no Markdown, so 'extraction at volume' describes that harness and not an archive writer. The seed path is not a code path in these checkouts.
- Replay is an audit view, not a rendering. It runs no JavaScript and writes no WACZ or rrweb file, and its chrome-dropped count is always zero (webfold/src/cli/replay.rs:370). Its raw HTML is cut at 50 KiB unless --full-html is set (webfold/src/cli/replay.rs:24).
Compared
How does it compare with other tools?
| Compared with | What the evidence says | Source |
|---|---|---|
| warcio 1.8.2 (Python, Apache-2.0) | warcio's PyPI description says it reads and writes WARC and ARC files, and is compliant with WARC 1.0 and 1.1. It is a library, not a capture tool. No fidelity or throughput comparison with webfold has been run. | warcio on PyPI |
| GNU wget --warc-file | wget writes WARC for a download when given --warc-file. webfold capture fetches one URL per command. No fidelity comparison has been run. | GNU Wget manual |
| Heritrix 3.18 (Apache-2.0) | Heritrix's README calls it the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler, and says it is designed to respect robots.txt. Webfold capture does not claim web-scale crawling. Neither has been compared with the other. | Heritrix README |
| Playwright, Puppeteer and Crawl4AI | The READMEs of Playwright, Puppeteer and Crawl4AI do not mention WARC (we searched each for the word on 2026-11-12); we have not tested whether they can write one. webfold has capture and replay. | Playwright README; Puppeteer README; Crawl4AI README |
Where should I read next?
FAQ
Frequently asked questions
Why is each record a separate gzip member?
So one record can be decoded from its recorded offset and length, without reading the records before it. Whole-file readers must decode every member, which MultiGzDecoder does.
Does capture follow the WARC standard?
It writes WARC/1.1 records with standard digest headers by default, and the reader accepts WARC/1.0 and WARC/1.1. The repository has no conformance test against a stock reader, so compliance is unverified.
Can replay show a page with its JavaScript?
No. replay writes one HTML file with each record's metadata, the webfold Markdown and the raw HTML, truncated at 50 KiB unless --full-html is given. It does not render the page or write WACZ or rrweb.
Are my credentials stored in the archive?
Request Authorization, Proxy-Authorization, Cookie and Cookie2 values are replaced with [REDACTED] by default, and the header names are kept. Response Set-Cookie headers are written as received. Turning redaction off writes secrets verbatim.
How many records does one capture write?
Four: warcinfo, request, response and metadata. The sidecars add one .cdx line for the response and one .idx line per record.
Can I extract text from a large archive?
webfold warc is the throughput harness. It prints counts and speed and writes no Markdown. Use a raw .warc with --threads for parallel reads; a .warc.gz runs on one thread through an external gunzip.
Read WARC with Webfold.
Run the command-line tool today, and join the waitlist for the hosted API.