Skip to content
Webfold Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Corpusforge

A corpus-governance pipeline: extract, scan and redact, build a deduplicated corpus and record an AI bill of materials.

Corpusforge is one command-line tool for the path from raw sources to a corpus you can govern: extract, scan and redact, ontology, corpus, adapt, bill of materials.

Commands

Command Does
corpusforge scan <path> Find secrets and personal data, offline. Exits 2 if any are found, so it gates CI
corpusforge redact <in> [-o out] Replace every detected span with [ENTITY_TYPE]
corpusforge extract <args> Extract text. PDFs run natively; other formats go through the extractor
corpusforge ontology <args> Tag content with the ontology tagger
corpusforge corpus <in> <out> [--target F] Chunk, deduplicate and select a coreset
corpusforge adapt <traces> <out.jsonl> --kind K --name N [--input-field F] Turn agent trace files into redacted, labelled rows
corpusforge aibom [-o out] Generate an AI bill of materials
corpusforge pipeline <src> <out> Extract, scan, redact, build the corpus and write the bill of materials, end to end 
corpusforge pipeline ./sources ./out
corpusforge adapt ./auto-traces ./router-witnesses.jsonl --kind tool_call --name route --input-field prompt --target 0.25

The adapter writes rows with text, class, label, kind, name and trace_id. Repeated inputs with conflicting labels are dropped instead of guessed.

Identity terms

scan and redact also look for names you supply. Set PII_IDENTITY_FILE (one term<TAB>ENTITY_TYPE per line) or PII_IDENTITY_TERMS.

Build

Corpusforge links the PII library as a path dependency, so check out pii-svc beside it:

<workspace>/
  pii-svc/         # provides pii-core
  corpusforge/

Hosted API

Two steps of the pipeline are also available over HTTP. POST /v1/corpus/redact redacts up to 100 records. POST /v1/corpus/build redacts if asked, splits each record into chunks of at most chunk_chars characters (preferring paragraph, line, then word boundaries) with optional overlap, and drops chunks whose normalised text was already seen. The rest of the pipeline, including the bill of materials, stays in the command line tool.

curl -s https://api.webfold.ai/v1/corpus/build \
  -H "Authorization: Bearer $WEBFOLD_API_KEY" -H "Content-Type: application/json" \
  -d '{"records":[{"id":"a","text":"..."}],"chunk_chars":2000,"overlap":200,"dedup":true,"redact":true}'