---
title: "Crawling and indexing"
description: "Crawl sites, fold shared chrome once, build a local index and search it."
---

## Crawl

```sh
webfold crawl <dir|url> [--min n] [--max n] [--chunk n]
```

`crawl` reads each page once, counts how many pages each block signature appears on, and omits blocks whose frequency crosses the template threshold. The report states how many blocks were shared chrome and how many content blocks were kept.

:::note
Many pages already lose their `<nav>` and `<footer>` during extraction, before the template pass runs. On such sites the report can show 0% shared chrome. That is a correct result, not a fault.
:::

## Index

```sh
webfold index crawl   <seed-url> [--max n] [--depth n] [--out file] [--http-cache dir]
webfold index search  <seed-url> <query> [--max n] [--depth n] [--top n]
webfold index map     <seed-url> [--max n] [--no-crawl-fallback]
webfold index resolve <guessed-url> [--max n]
```

| Command | Does |
| --- | --- |
| `index crawl` | Crawl from a seed and serialise the extracted pages as JSONL. |
| `index search` | Crawl from a seed, build the in-memory index, and rank pages for a query. |
| `index map` | Discover same-origin URLs: sitemap first, then a bounded crawl fallback when the sitemap is thin or missing. |
| `index resolve` | Guess a topic URL on an unstable host (for example a government site) and return the canonical one. |

Ranking uses a term-frequency score: query-term boundary matches normalised by the log of page length. It is a lexical index, not an embedding index.

## Offline search over saved HTML

```sh
webfold search "rotate wal segment" --corpus ./saved-pages --count 10
webfold folder-search . "rotate wal segment" -n 5
```

`--corpus` indexes a directory of saved HTML with no live fetch. `folder-search` searches a local folder; add `--semantic` with `WEBFOLD_EMBED_BASE_URL` set to use an embedding endpoint you run.

## Batch and chunk

```sh
webfold batch ./dir-of-html-files
webfold chunk page.html --size 4000 --jsonl
webfold sink jsonl < records.ndjson     # also: parquet, iceberg
```

`chunk` accepts HTML, Markdown, text, PDF or a URL and emits retrieval-ready chunks. `sink` writes NDJSON records to JSONL, Parquet or Iceberg.