Overview
What Webfold is, what it does, and how the pieces fit together.
Webfold is a Rust library and command-line tool for converting HTML and documents to Markdown, removing repeated blocks across pages, and running crawl and browser workflows. It turns messy pages into clean text that people and AI systems can use.
What it does
- Extract. Convert HTML to Markdown blocks, resolve links, and drop navigation and footer chrome that repeats across pages.
- Acquire. Fetch URLs natively, fall back to a rendered page when the content only exists after JavaScript runs, and capture responses to WARC.
- Crawl and index. Walk a site, fold shared chrome once, and search the extracted pages with a local term-frequency index.
- Search. Normalise results from web and academic search engines into one result shape.
- Convert documents. PDF, DOCX, XLSX and CSV to Markdown through the Superdocuments converter.
- Serve agents. An MCP server (
webfold-mcp), a Chrome DevTools Protocol (CDP) subset server for Playwright and Puppeteer, and an intercepting proxy.
Ways to use it
| Surface | Use it when | Start here |
|---|---|---|
CLI (webfold) |
You want clean Markdown from a URL or file in a script or CI job. | Quickstart |
| Rust library | You want conversion inside your own program, with no separate process. | Library |
MCP server (webfold-mcp) |
An AI agent needs to fetch, search and crawl. | MCP server |
| CDP server | You have Playwright or Puppeteer scripts and want a lightweight endpoint. | Headless and CDP |
| Documents and OCR | You need Word, Excel, PDF or scanned files as Markdown. | OCR and documents |
| PII and secrets | You need to find and redact personal data and credentials. | PII and secrets |
| Corpusforge | You are building a governed training or search corpus. | Corpusforge |
| Glossa | You need to localise a product. | Glossa |
| Computer use | An agent needs a desktop that a person can supervise. | Computer use |
Hosted API (api.webfold.ai) |
You want the same capabilities over HTTP with an API key and no infrastructure. | API overview |
How extraction works
Webfold scans the HTML into Markdown blocks and gives each block a 64-bit signature. Across the pages of one site, blocks that appear on most pages (navigation, footers, sidebars) are classed as template chrome and omitted from the output. Recurrence is a heuristic: repeated valid article content can also be removed.
Project boundaries
Webfold owns HTML extraction and rendering, URL acquisition, crawling, browser and CDP workflows, search interfaces, WARC capture and replay, and the CLI and MCP integration. Structured document conversion (PDF, Office formats) delegates to Superdocuments.
License
Webfold is licensed under Apache 2.0. The source repository is private and opens with the public release.