Scrape a sitemap to Markdown

Turn a sitemap or a list of URLs into a folder of Markdown files, with a manifest, a failure ledger and resume.

bulk is the whole-site job: read a sitemap (or a file of URLs), pick the pages you want, fetch them concurrently, extract the main content of each, and write one Markdown file per page with a record of what worked and what did not.

webfold bulk <sitemap-url|feed-url|urls-file>... --out DIR
       [--include PREFIX]... [--limit n] [--seed n | --no-shuffle]
       [--concurrency n] [--delay-ms n] [--timeout-ms n] [--resume] [--http-cache DIR]

Example: a documentation site

webfold bulk https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz \
  --include https://developer.mozilla.org/en-US/docs/ \
  --limit 1000 --concurrency 64 --out mdn

This reads the gzipped sitemap, keeps URLs under /en-US/docs/, shuffles them with seed 42, takes the first 1,000, fetches 64 at a time and writes the results into mdn/.

What you get

File	Contents
`NNNNN.md`	The page as Markdown, numbered by position in `urls.txt`.
`manifest.csv`	`index,url,file,status,html_bytes,markdown_bytes,fetch_ms,title` for every saved page.
`failures.csv`	`index,url,error` for every page that was not saved.
`summary.json`	Counts, bytes, concurrency and elapsed time.
`urls.txt`	The selected URL list, so a re-run uses exactly the same pages.

A page counts as saved only when it came back with HTTP 200, looked like HTML, and produced content. An HTTP 200 page with nothing extractable is a failure with the reason http 200 but no content extracted. “Fetched” is not “saved”, so check failures.csv as well as the summary.

Selection

Sources. A sitemap, a sitemap index (one level of child sitemaps, up to 64), an RSS or Atom feed, a gzipped sitemap, or a text file with one URL per line.
--include. Keep only URLs that start with the prefix. Repeat it for several prefixes.
--limit and --seed. URLs are shuffled with a seeded generator before the limit is applied, so the sample is spread across the site and the same seed gives the same pages. --no-shuffle keeps sitemap order.

Concurrency and politeness

--concurrency (default 32) bounds requests in flight. --delay-ms sets a minimum gap between requests to the same host (default 0). The fetcher honours robots.txt (a disallowed page is listed in failures.csv as blocked by robots.txt), reuses connections, retries a transport error, a 429 or a 5xx once, and enforces --timeout-ms (default 30,000) per page. Start with a modest concurrency against a site you do not run, and raise --delay-ms if the site asks.

Webfold does not rotate IP addresses or solve challenges. A site that blocks one machine’s requests will block bulk, and those pages will appear in failures.csv.

Resume

webfold bulk --out mdn --resume

With --resume, bulk reads mdn/urls.txt and skips every index whose NNNNN.md already exists and is not empty. Delete a file to fetch that page again.

What it does not do

HTML pages only. A PDF, image or other type is listed in failures.csv as not html: <content type>. Use webfold convert or webfold pdf on those.
No JavaScript. Pages whose content exists only after scripts run come back thin or empty. Use rendered pages for those.
Extraction is main-content selection, not a fidelity guarantee. Some boilerplate survives: on MDN pages, a “Baseline” status box and the “Help improve MDN” line came through in the sample we read.
Single machine. There is no distributed worker pool. Throughput is what one process on your host and network gives you.

Measured

Live, MDN’s documentation sitemap: 7,000 pages sampled with seed 42, 96 at a time, on a 4-core machine: 7,000 of 7,000 saved in 16.4 seconds (about 427 pages a second), 1.27 GB of HTML down to 34.7 MB of Markdown, no failures. A second sample (seed 7, about half the pages new) took 15.8 seconds.
Webfold alone, over loopback: 7,000 synthetic documentation pages of about 10 KB each (73 MB of HTML) in 0.8 to 0.9 seconds. This is the local cost only, with no network.

The first version of the command took 282 seconds for the same 7,000 pages. Every request asked the server to close the connection, so each page paid a TCP and TLS handshake, and the destination check looked the host up again for every request. Reusing connections and caching the resolved answer for 30 seconds fixed it.

The live figure depends on this machine’s connection and on the site at the time; one run per sample, one day. See the benchmark for the method and limits.