Crawl a web site

BFS crawl starting from url, staying on the same origin, up to max_pages and max_depth. Each page is stored in the local web index and returned as Markdown.

POST/v1/web/crawl

Authorization

AuthorizationBearer token (JWT) · headerrequired

Bearer API key issued for your account. Send Authorization: Bearer <key>.

Request body

requiredapplication/json

urlstring<uri>required

max_pagesinteger

Maximum number of pages to fetch

default: 10

max_depthinteger

Maximum link depth from the seed

default: 2

Responses

200

Crawled pages

pagesWebCrawledPage[]required

Show properties

Array of WebCrawledPage

urlstring<uri>required

titlestringrequired

markdownstringrequired

400

Bad request

string

401

Missing or invalid bearer token

string

403

Billing or plan entitlement blocked the request

string

502

Upstream service (arXiv, YouTube, web page) failed

string

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Bearer token (JWT)Remember on this deviceInclude my values in samples

Bodyapplication/json

urlstring*max_pagesintegerMaximum number of pages to fetchmax_depthintegerMaximum link depth from the seed

Request

curl -X POST 'https://api.webfold.ai/v1/web/crawl' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://example.com",
  "max_pages": 10,
  "max_depth": 2
}'

import requests

response = requests.post(
    "https://api.webfold.ai/v1/web/crawl",
    headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"url\": \"https://example.com\",\n  \"max_pages\": 10,\n  \"max_depth\": 2\n}",
)

const response = await fetch("https://api.webfold.ai/v1/web/crawl", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"url\": \"https://example.com\",\n  \"max_pages\": 10,\n  \"max_depth\": 2\n}"
});

Response

{
  "pages": [
    {
      "url": "http://example.com",
      "title": "string",
      "markdown": "string"
    }
  ]
}

"string"

"string"

"string"

"string"