Product · Crawl
A whole site, without the same menu on every page.
Crawl a site or a folder of saved pages. Webfold keeps what is different on each page. Navigation that repeats is counted once. Footers that repeat are left out.
See it work
Try it from wherever you work.
1webfold crawl https://example.com --max 50
2WEBFOLD_EMIT=1 webfold crawl ./saved-site
What you get
What Crawl does.
Repeated chrome removed
Each block of text gets a signature. Blocks that appear on most pages of a site are treated as page furniture and dropped.
A report, not a guess
The crawl tells you how many blocks were shared furniture and how many were content, so you can see what was removed.
Crawl the web or a folder
Point it at a URL or a directory of saved HTML. Offline work needs no network.
Index as you go
`index crawl` writes pages to JSONL, and `index search` ranks them for a query, with an on-disk HTTP cache to save repeat fetches.
How it works
Three steps.
- 1
Discover
Follow links on the same site up to your page and depth limits.
- 2
Count
Count each block signature once per page across the set.
- 3
Keep what differs
Output the content blocks and report the shared ones.
Limits up front.
Know the limits
- Repetition is a heuristic: valid content that repeats across pages can also be removed.
- It needs several pages from the same template to learn what the furniture is.
- Many pages already lose their menus during extraction, so on some sites the shared-furniture figure is 0%. That is a correct result.
Keep reading.
FAQ
Questions.
Does it respect robots.txt?
Webfold's harness fetch applies a robots check, and polite crawling settings are documented. Check your build's settings before crawling a site you do not own.
How many pages can it crawl?
You set the limit with --max or max_pages. Large archive extraction is covered under Archive.
Use Crawl with Webfold.
Run the command-line tool today. Join the waitlist for the hosted API.