Webfold
Menu

Benchmark · Measured

Web archive extraction

Webfold extracts clean text from an 84,749-page Common Crawl subset. On 8 threads it does 26,675 pages a second, peaking at 4.8 GB. On one thread it does 2,943 a second in 35 MB. These are different configurations. The seed path drops 134 pages at block boundaries.

Clean-text extraction over a Common Crawl subset, with each configuration named.

Run: 2026

What question does this benchmark ask?

How many archived pages a second can Webfold extract? What does that cost in memory?

How was it run?

  • Corpus: an 84,749-page Common Crawl subset (15.9 GB).
  • Each row is a different configuration. Do not quote one as a single headline.

What are the results?

Pages per second by configurationLonger is better
Seed path, 8 threads
26,675
Raw WARC, 8 threads
14,238
Seed path, 1 thread
2,943
Peak memory by configuration, megabytesShorter is better
Seed path, 1 thread
35 MB
Seed path, 8 threads
4.8 GB
Raw WARC, 8 threads
15.2 GB
ConfigurationPages per secondMemory
Seed path, 8 threads26,6754.8 GB peak
Raw WARC, 8 threads14,23815.2 GB peak
Seed path, 1 thread2,94335 MB peak

Pick a point on the memory curve: lowest memory on one thread, highest throughput on eight.

What does this not show?

Know the limits

  • The seed path drops 134 of the 84,749 pages at block boundaries; the raw path keeps all of them but holds the corpus in memory.
  • One host, one corpus. The README's separate 16-thread raw figure (10,597 pages per second) is a different configuration.

Source: research/syndesi-swarm-webfold, section 6.3

Which other benchmarks are there?

Check it on your own pages.

Run the command-line tool on your content.

Join the waitlistRead the quickstart