Webfold
Menu

Benchmark · Directional

A whole site to Markdown

One 4-core machine converted 7,000 of 7,000 live MDN pages to Markdown. It took about 16 seconds. None failed. A single machine does not rotate addresses or solve challenges. Each sample is one run on one day.

A sitemap in. One Markdown file per page out. 7,000 live MDN pages in 16 seconds. The first run was 17 times slower. Two per-request costs caused it.

Last updated , the newest measurement or check on record for this page.

Run: 9 November 2026

What question does this benchmark ask?

How fast can one process turn a sitemap into Markdown files? How does that compare with a hosted batch service?

How was it run?

  • Command: webfold bulk <sitemap> --out DIR. It reads the sitemap, shuffles the URLs with a fixed seed, takes a limit, fetches with bounded concurrency, extracts the main content, and writes NNNNN.md, manifest.csv, failures.csv and summary.json. robots.txt is honoured.
  • Live run: pages sampled from MDN's English documentation sitemap (the one used in a KDnuggets tutorial for the hosted service Olostep), 96 requests at a time, from a 4-core machine over its internet connection.
  • Loopback run: 7,000 and 10,000 synthetic documentation pages of about 10 KB each, served from the same machine, to isolate Webfold's own cost from the network.

What are the results?

Live MDN, 7,000 pages, pages per secondLonger is better
Second sample (seed 7)
443
First sample (seed 42)
427
First version, before the fix
24.8
Live MDN, 7,000 pages, total time, secondsShorter is better
Second sample (seed 7)
15.8 s
First sample (seed 42)
16.4 s
First version, before the fix
282 s
RunPagesResult
Live MDN, seed 42, 96 at a time7,000 (1.27 GB of HTML)saved 7,000, failed 0, 16.4 s (426.6 pages/s)
Live MDN, seed 7, 96 at a time7,000 (3,675 not in the first sample)saved 7,000, failed 0, 15.8 s (443.3 pages/s)
Live MDN, first version of the command7,000saved 6,996, failed 4 (connection closed), 282.2 s (24.8 pages/s)
Loopback, 96 at a time7,000 (73 MB of HTML)saved 7,000, failed 0, 0.91 s (7,718 pages/s)
Loopback, 96 at a time10,000 (104 MB of HTML)saved 10,000, failed 0, 1.15 s (8,703 pages/s)

One 4-core machine saved 7,000 of 7,000 MDN pages in about 16 seconds, 1.27 GB of HTML down to about 35 MB of Markdown, with no failures, on two different samples.

The first version took 282 seconds for the same job. Profiling found two per-request costs. Every request asked the server to close the connection, so each page paid a TCP and TLS handshake, and the destination check looked the host up again for every request (this machine's resolver answered about 25 lookups a second). Reusing connections and caching the resolved answer for 30 seconds took the same job from 282 s to 16 s.

For scale, the tutorial for the hosted service reports 7,225 of 10,000 pages in 2 minutes 52 seconds (about 42 pages a second) and all but 62 in about 22 minutes overall. Those are its figures on its own infrastructure and a different page sample; we did not run that service.

What does this not show?

Know the limits

  • The live figures depend on this machine's connection and on the site's response at the time: one run per sample, on one day. MDN is served from a CDN, and the first sample had been fetched by this machine before; the second sample, where about half the pages were new, ran at the same speed.
  • A single machine does not rotate addresses or solve challenges, which the hosted service sells. A site that rate-limits one address will slow this down. The loopback figure measures Webfold only and is a ceiling, not a prediction.
  • Output quality is not scored here. On the MDN pages we read, the main content came through, and some boilerplate did too (a Baseline status box and the Help improve MDN line). HTML pages only: a PDF is listed in failures.csv, not converted, and pages that need JavaScript to appear come back thin.
  • Connection reuse is on for bulk only. The general fetcher still sends Connection: close, because a connection pooled by one runtime is unusable on another; the fix is in the bulk path, which makes all its requests on one runtime.
  • Comparison figures for the hosted service are quoted from its tutorial; we did not run that service.

Source: webfold: examples/bulk_bench.rs and tests/bulk_local.rs; live runs of webfold bulk on 2026-11-09; KDnuggets, Web Scraping 7,000 Pages in 3 Minutes with Python and Olostep

Which other benchmarks are there?

Check it on your own pages.

Run the command-line tool on your content.

Join the waitlistRead the quickstart