Chunk and de-duplicate records into corpus rows

Optionally redacts each record first, then splits it into chunks of at most chunk_chars characters (preferring paragraph, line then word boundaries) with optional overlap, and drops chunks whose normalised text was already seen. Offsets are character offsets into the (possibly redacted) record text. Each record is metered, as corpus.build.

POST/v1/corpus/build

Authorization

AuthorizationBearer token (JWT) · headerrequired

Bearer API key issued for your account. Send Authorization: Bearer <key>.

Request body

requiredapplication/json

recordsCorpusRecord[]required

min items 1 · max items 100

Show properties

Array of CorpusRecord

idstring

Optional; defaults to rec-<n>.

textstringrequired

chunk_charsinteger

min 256 · max 20000 · default: 2000

overlapinteger

At most half of chunk_chars.

min 0 · default: 0

dedupboolean

default: true

redactboolean

Redact each record before chunking.

default: false

Responses

200

Corpus rows and statistics

rowsobject[]required

Show properties

Array of object

idstringrequired

chunk_indexintegerrequired

startintegerrequired

Character offset.

endintegerrequired

hashstringrequired

Hash of the normalised text; equal hashes are duplicates.

textstringrequired

statsobjectrequired

Show properties

records_inintegerrequired

chunks_totalintegerrequired

rows_outintegerrequired

duplicates_droppedintegerrequired

redactedbooleanrequired

findings_totalintegerrequired

400

Bad request

string

401

Missing or invalid bearer token

string

403

Billing or plan entitlement blocked the request

string

502

Upstream service (arXiv, YouTube, web page) failed

string

503

The backing service is not configured or is busy

string

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Bearer token (JWT)Remember on this deviceInclude my values in samples

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.webfold.ai/v1/corpus/build' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "records": [
    {
      "id": "string",
      "text": "string"
    }
  ],
  "chunk_chars": 2000,
  "overlap": 0,
  "dedup": true,
  "redact": false
}'

import requests

response = requests.post(
    "https://api.webfold.ai/v1/corpus/build",
    headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"records\": [\n    {\n      \"id\": \"string\",\n      \"text\": \"string\"\n    }\n  ],\n  \"chunk_chars\": 2000,\n  \"overlap\": 0,\n  \"dedup\": true,\n  \"redact\": false\n}",
)

const response = await fetch("https://api.webfold.ai/v1/corpus/build", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"records\": [\n    {\n      \"id\": \"string\",\n      \"text\": \"string\"\n    }\n  ],\n  \"chunk_chars\": 2000,\n  \"overlap\": 0,\n  \"dedup\": true,\n  \"redact\": false\n}"
});

Response

{
  "rows": [
    {
      "id": "string",
      "chunk_index": 0,
      "start": 0,
      "end": 0,
      "hash": "string",
      "text": "string"
    }
  ],
  "stats": {
    "records_in": 0,
    "chunks_total": 0,
    "rows_out": 0,
    "duplicates_dropped": 0
  },
  "redacted": true,
  "findings_total": 0
}

"string"

"string"

"string"

"string"

"string"