Product · Corpusforge
From raw sources to a governed training corpus.
One command-line pipeline builds a governed corpus. It extracts the text. It redacts it. It deduplicates the result. It records what went into it.
See it work
Try it from wherever you work.
1corpusforge pipeline ./sources ./out
What you get
What Corpusforge does.
Scan and redact, offline
Secrets and personal data are found and replaced with no network call. `scan` exits non-zero when it finds something, so it gates CI.
Corpus building
Chunking, deduplication and coreset selection run natively, to shrink a corpus without losing coverage.
Trace adapters
Turn agent trace files into redacted labelled rows. Repeated inputs with conflicting labels are dropped, not guessed.
A bill of materials
Generate an AI bill of materials for the corpus you built.
How it works
Three steps.
- 1
Extract
PDFs natively; other formats through the document converter.
- 2
Scan and redact
Find and replace personal data and secrets.
- 3
Build and record
Chunk, deduplicate, select a coreset, then write the bill of materials.
Limits up front.
Know the limits
- Some stages (non-PDF extraction, ontology tagging, bill of materials) still call external tools.
- Deduplication today removes exact duplicate chunks by comparing strings, which gets slow on very large corpora (the cost grows with the square of the chunk count); a hash-based pass is next.
- Known gaps include Common Crawl ingestion, a crawl frontier, near-duplicate detection beyond the current method, shard output, language identification, tokenisation and quality scoring.
- It builds against the PII library as a sibling checkout, so it is source-only for now.
Keep reading.
FAQ
Questions.
Is it a hosted service?
The full pipeline is a command-line tool you run on your own machine. The hosted API also has two endpoints for the common steps: /v1/corpus/redact redacts a batch of records, and /v1/corpus/build chunks and de-duplicates them.
Use Corpusforge with Webfold.
Run the command-line tool today. Join the waitlist for the hosted API.