Benchmark · Directional
Folder search
Webfold's folder search put the right file first more often than xerj. The score is 0.946 against 0.808. It is merged to the main branch, in no release yet. Queries are cut from the folders themselves. It is lexical ranking only.
Webfold's local search against xerj. Two folders: 554 files, 1,026 files. A held-out query set.
What question does this benchmark ask?
Queries are plain words cut from a folder. How often is the right file first? What does it cost to index? To query? To keep resident?
How was it run?
- Two corpora: Webfold's own docs, source and tests (554 files indexed) and the superdocuments repository's text files (1,026 files). Queries are cut from the corpus so the right file is known: prose lines and code lines, each as a phrase and as keywords. The first sampler is the one of the 4 November run (240 queries, six-word phrases, the three rarest words); the second is held out, built differently (four-word phrases, three random words), with a different seed (188 and 203 queries). Nothing was tuned on the held-out set.
- Run on a shared 4-core VM at a load average of 12 to 32. xerj v1.0.0-rc.100 from its installer, default lexical ranking, no embeddings, one fresh node per corpus. Webfold is the folder search as merged to its main branch on 13 November (a memory-mapped positional index, phrase and proximity ranking, and a serve mode). No release includes it yet. The run used a build from before the last rebase, which changed only dependency pins. The before column is the same code before that change.
- Per-process rows time a new process per query (xerj's search command is a client of a running node, so its node is extra); shared-process rows time Webfold's serve mode and the xerj node over one connection.
What are the results?
| Measurement | Webfold folder-search | xerj |
|---|---|---|
| hit@1, Webfold's files, 240 queries | 0.946 | 0.808 (node), 0.804 (search command) |
| hit@5 / mean reciprocal rank | 1.000 / 0.971 | 0.950 / 0.875 |
| Phrase queries on prose, hit@1: Webfold's files / held-out / second corpus | 1.000 / 0.917 / 0.867 | 0.983 / 0.817 / 0.717 |
| Keyword queries on code, hit@1 (Webfold's files) | 0.917 | 0.617 |
| hit@1, held-out queries (188) / second corpus (240) / second corpus held-out (203) | 0.931 / 0.846 / 0.926 | 0.739 / 0.683 / 0.778 |
| Per query, new process each time (median, Webfold's files / second corpus) | 4.0 / 4.9 ms | 2,200 / 162 ms (search command) |
| Per query, running service (median, first / second corpus) | 0.7 / 0.6 ms (serve) | 3.4 / 5.9 ms (node) |
| Index build (first / second corpus) | 0.42 / 0.40 s | 46.6 / 41.1 s (8.7 / 5.0 s of CPU) |
| Index on disk (first / second corpus) | 4.8 / 2.4 MB | 28.4 / 15.9 MB |
| Memory: a query process / the long-lived process | 20 MB peak / serve 16 MB | 99 MB client / node 468 to 516 MB (997 MB peak while indexing) |
| Cold start, files evicted (Webfold's files) | 10.7 ms (a process); serve ready in 9.8 ms, first query 2.6 ms | 4.5 ms (running node, its own caches kept); node restart 0.85 s |
On both folders and both query sets Webfold put the right file first more often than xerj in every category we split out, including phrases on prose, which xerj won on 4 November (0.917 against 0.783). Before the change the same code scored 0.733 on prose phrases on these files; after it, 1.000 (held-out 0.917, against xerj 0.817).
A Webfold process per query takes about 4 to 5 ms and 13 to 20 MB; with serve mode a query takes under a millisecond in about 12 to 16 MB. The xerj node answered in 3 to 6 ms and holds 290 to 520 MB.
The one cell xerj wins is a cold start on Webfold's own files: its running node answered in 4.5 ms with its data files evicted, a Webfold process in 10.7 ms. They are not the same measurement (the node keeps its own caches), and a cold Webfold serve answered its first query in 2.6 ms.
What does this not show?
Know the limits
- Two folders on one machine, one run per cell, at a shared load. The held-out set is a second sampler over the same files, not different files. Lexical ranking only; xerj's neural and hybrid modes were not tried.
- Queries are cut from the corpus, which favours engines that match text exactly, and Webfold's phrase ranking is a positional exact-match boost. The first folder is not the 483-file snapshot of the 4 November run, so the old and new numbers are not on the same files; the before column was re-run on these.
- xerj's search command spent about 0.9 s of CPU per query on the first folder and 26 ms on the second; the first run saw 127 ms. We could not find why, so the second folder's 162 ms is the fairer figure for that command.
- On the second folder 3 of 240 and 1 of 203 xerj requests could not be captured and count as misses (at most 1.3 percentage points). Prose keyword queries on the second folder are low for both (0.583 and 0.400).
- This compares file search only. It says nothing about fetching, rendering or extraction.
Which other benchmarks are there?
- HTML to Markdown conversion Measured
- Document conversion Measured
- Main-content extraction quality Measured
- Web archive extraction Measured
- Footprint of a scrape or crawl Directional
- PII detection against Presidio Measured
- Localisation extraction Directional
- PDF text extraction with OCR Directional
- PDF extraction against other extractors Directional
- LaTeX conversion with BibTeX Directional
- A whole site to Markdown Directional
- Agent-browser compatibility (Lexbench) Measured
- Headless engine over CDP Directional
- Search-results parsing Directional
Check it on your own pages.
Run the command-line tool on your content.