Webfold
Menu

Benchmark · Measured

PII detection against Presidio

terredactal is Webfold's Rust PII detector. On 2,000 generated documents it scores F1 0.998 against 0.867 for Presidio's defaults. It runs at 110,200 documents a second against 80. We wrote the corpus. Names need a slower language-model tier.

terredactal against Microsoft Presidio on three kinds of test set.

Last updated , the newest measurement or check on record for this page.

Run: 11 to 13 November 2026

What question does this benchmark ask?

Is terredactal faster than Microsoft Presidio? Is it lighter? Is it as accurate? The same scorer scores both. The language-model tier is tested both ways.

How was it run?

  • 2,000 generated documents (681,518 bytes, six genres: e-mails, support tickets, clinical notes, invoices, logs, config files), fixed seed, 10,398 labelled spans of 13 types (6,686 over the 11 pattern types, 2,773 names, 939 places), and nine kinds of lookalike that must not be flagged (numbers that fail their checksum, UUIDs, git hashes, version strings, timestamps).
  • Presidio 2.2.366 with spaCy en_core_web_lg, run three ways: its defaults, with its Australian recognizers switched on, and with its language model removed (patterns only). For names and places Presidio was also run with its transformers configuration (StanfordAIMI/stanford-deidentifier-base). terredactal: the default detector tier (patterns, checksums, context), and with the optional language-model tier on (GLiNER-PII, knowledgator/gliner-pii-base-v1.0).
  • Names and places: 244 sentences written by hand before the model tier was run on them (192 names, 211 places), and 33 e-mail, ticket and form layouts written after terredactal's layout rules were fixed (22 names, 7 places). Public samples: CoNLL-2003, ai4privacy 300k and 400k, Gretel, SPY and TAB, converted with a label mapping written down before any tool ran on them.
  • Speed and memory with the model tier on and off: the first 300 corpus documents, all tools in the same session, on a machine that was also running other jobs (load average 8 to 15 on four cores), so wall-clock figures are pessimistic for every tool.
  • One scorer for both: precision, recall and F1 by exact span and by relaxed span (at least half the labelled span, same type), over the types both tools support. Four cores of a shared 2.3 GHz Xeon; medians of repeated sweeps; Presidio opened no network connections.

What are the results?

Documents scanned per secondLonger is better
terredactal
110,200
Presidio, patterns only
235
Presidio, defaults
80
Presidio, Australian set
77
Peak memory, megabytesShorter is better
terredactal
13 MB
Presidio, patterns only
135 MB
Presidio, defaults
781 MB
Presidio, Australian set
781 MB
False alarms on the lookalike numbers (lower is better)Shorter is better
terredactal
17
Presidio, defaults
525
Person names on 244 hand-written sentences, F1 by exact spanLonger is better
terredactal, language-model tier
0.960
Presidio, spaCy
0.857
Presidio, transformers
0.821
Documents per second with the language-model tier on (300 documents)Longer is better
Presidio, spaCy1,043 MB
52
Presidio, transformers1,487 MB
3.4
terredactal, language-model tier on1,167 MB
1.9
System (types compared)F1, relaxedF1, exactFalse alarmsDocuments/sPeak memoryCold start
terredactal (7 types shared with Presidio's defaults)0.9980.99817110,20013 MB16 ms
Presidio, defaults0.8670.77852580781 MB2.20 s
terredactal (11 types shared with the Australian set)0.9970.99735110,20013 MB16 ms
Presidio, Australian recognizers on0.9240.85256177781 MB2.11 s
Presidio, patterns only (language model removed)0.9250.855783235135 MB1.18 s
terredactal with the language-model tier on (9 types shared with Presidio's defaults, adds names and places)0.9830.983171.9 (300 documents)1,167 MB3.0 s
Presidio, defaults (the same 9 types; speed and memory measured with its Australian recognizers on)0.8760.77858352 (300 documents)1,043 MB7.7 s

The multiples on the home page and the benchmarks page are two rows divided. 110,200 against 235 documents a second is 470 times, with Presidio's language model removed. 110,200 against 80 is about 1,400 times, against its defaults.

On the patterns both tools recognise, terredactal finds essentially everything (recall 1.000) and raises about one thirtieth of Presidio's false alarms, because it checks the checksums and the context that separate a real number from a lookalike. It does this at 110,200 documents a second in 13 MB, against 80 documents a second in 781 MB for Presidio's defaults, or 235 a second in 135 MB with Presidio's language model removed.

Names and places need a language model, so they are not in the pattern rows. With terredactal's optional language-model tier on, its exact-span F1 for names on the 244 hand-written sentences is 0.960 against 0.857 for Presidio's spaCy model and 0.821 for its transformers model, and for places 0.974 against 0.884 and 0.827. On the 33 e-mail and ticket layouts, whose layout rules we wrote after seeing where the model missed, names reach 0.978 against 0.727 and 0.600, on 22 names.

On the public samples the same tier has the higher exact-span F1 for names on CoNLL-2003 (0.854 against Presidio's spaCy 0.762), Gretel (0.727 against 0.579) and both ai4privacy samples (0.667 against 0.404 on the 400k sample, 0.510 against 0.217 on the 300k sample). Those comparisons are against Presidio's spaCy model; Presidio's transformers model, run on the first 100 Gretel documents only, scored higher for names there (0.712 against 0.689). It is behind on place names in CoNLL-2003 (0.701 against 0.775) and, with its layout rules, gives up a little name precision on the two ai4privacy samples (exact F1 0.510 with the rules against 0.527 without them on the 300k sample, 0.667 against 0.673 on the 400k sample).

The price is speed and memory. With the language-model tier on, terredactal handled about 2 documents a second in 1.2 GB on the 300-document subset, slower per CPU-second than Presidio's transformers configuration (4.1 documents per CPU-second against 2.1) and far slower than its spaCy configuration (59). The 110,200 documents a second is the patterns-only tier, which is a separate, optional-model-free configuration.

Australian company numbers (ACN): terredactal now has a recogniser, and it reports a nine-digit number only when the text says it is a company number. On 50 hand-written labels written after the rule was fixed, it found 0.860 of them with no false alarms (precision 1.000); Presidio found 0.900 of them at a precision of 0.634, because it reports bare nine-digit numbers that pass the check digit, such as order numbers.

What does this not show?

Know the limits

  • The corpus is generated by us, from templates. It measures these recognizers on these patterns and says nothing about real-world text. Names and addresses, the hard class, are the least representative part of it, and terredactal's layout rules for names were written after reading where the model missed names in it, so its name figures on that corpus are not evidence.
  • The two hand-written sets for names and places are short sentences and short e-mails written by the authors of the tool being tested. They are checks on short prose, not samples of real documents; the 33 e-mail layouts hold 22 names, so one name is 4.5 points of recall.
  • The public samples are small, seeded draws. TAB gold spans include titles such as Mr, so exact match is near zero for both tools there; SPY labels only some of the e-mails, numbers and URLs in each document, so precision there means little; many of ai4privacy's labelled card numbers fail the card checksum. Presidio's transformers configuration was run on the first 2 to 200 documents of each sample only, because it is slow. Per-corpus tables, error lists and the label mapping are in the repository.
  • Running it exposed span bugs in terredactal (an IBAN that included a trailing space, a URL that included a sentence-final period, a phone number that lost its opening bracket, Medicare numbers cut short) and a UK NHS pattern that claimed other numbers. We fixed them with unit tests and published the before and after tables next to the results. Presidio's numbers did not change between the two runs.
  • terredactal flags 18 eleven-digit business numbers as Medicare numbers because they pass the check digit by chance. The ACN rule was tuned in two rounds; only the first figure (0.860, on a set written after the rule was fixed) is a clean held-out result. A second round raised it to 0.960 on that same set after its errors had been read, which is not.
  • Presidio was run through its Python API with one document per call. A deployment that batches or scales out would change its speed figures, not its accuracy.
  • Run on a shared virtual machine with other jobs running; each measured run was sequential, background load was not controlled. The speed figures for the language-model tiers were taken on a busier machine than the pattern figures, and Presidio's transformers configuration was limited to one thread after a first attempt with the default threads stalled under the load.
  • Test it on your own data before you rely on any of this. The generator, scorer and reproduce.sh are in the terredactal repository under benchmarks/presidio-h2h.

Source: terredactal repository, benchmarks/presidio-h2h (README, results/results.md, results/perf.md)

Which other benchmarks are there?

Check it on your own pages.

Run the command-line tool on your content.

Join the waitlistRead the quickstart