{
"id": "batch_9f2c8a",
"status": "queued",
"mode": "scrape",
"format": "markdown",
"tags": [
"docs"
],
"crawl": {
"source": {
"type": "start_url",
"url": "https://example.com/docs"
},
"max_pages": 500,
"max_depth": 0,
"follow_subdomains": true,
"url_pattern": "^https://example\\.com/docs/"
},
"input": {
"reserved": 24817,
"reserved_is_ceiling": false,
"submitted": 25000,
"duplicates": 183,
"invalid": 0
},
"credits": {
"reserved": 24817
},
"created_at": "<string>",
"invalid_urls": [
{
"url": "<string>",
"reason": "Must be a public http:// or https:// URL"
}
],
"cache_metadata": {
"status": "hit",
"age_ms": 1
},
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"webhook_secret": "<string>",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"message": "<string>",
"error_code": "RATE_LIMITED",
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}Submit a batch
Scrape up to 25,000 URLs, or crawl a site, asynchronously. Poll the batch ID or receive a webhook when it finishes.
{
"id": "batch_9f2c8a",
"status": "queued",
"mode": "scrape",
"format": "markdown",
"tags": [
"docs"
],
"crawl": {
"source": {
"type": "start_url",
"url": "https://example.com/docs"
},
"max_pages": 500,
"max_depth": 0,
"follow_subdomains": true,
"url_pattern": "^https://example\\.com/docs/"
},
"input": {
"reserved": 24817,
"reserved_is_ceiling": false,
"submitted": 25000,
"duplicates": 183,
"invalid": 0
},
"credits": {
"reserved": 24817
},
"created_at": "<string>",
"invalid_urls": [
{
"url": "<string>",
"reason": "Must be a public http:// or https:// URL"
}
],
"cache_metadata": {
"status": "hit",
"age_ms": 1
},
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"webhook_secret": "<string>",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"message": "<string>",
"error_code": "RATE_LIMITED",
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}{
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"message": "<string>",
"error_code": "INTERNAL_ERROR",
"required_permission": "logs:read",
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}batches:write permission. See the guide for examples and usage.Authorizations
Send Authorization: Bearer <API_KEY>. Keys have full access unless restricted to scopes.
Headers
Unique key per submission. Retrying with the same key and body returns the original batch; a different body returns 409.
200Body
Create an asynchronous scraping job.
Scrape a list of up to 25,000 URLs.
- Scrape
- Crawl
Show child attributes
Show child attributes
Legacy URL notified when the batch finishes. Preserves one best-effort attempt. Cannot be combined with webhook.
Where to send the batch's final-status event. Omit retry for one attempt; {} uses the default retry schedule.
Show child attributes
Show child attributes
Tags stored on the batch. Filter the batch list by them later.
201 - 50["docs", "competitor"]
Response
Batch accepted. Read progress and results from GET /batch/{batch_id}.
Batch ID. Poll GET /batch/{batch_id} with it.
"batch_9f2c8a"
Always queued. An accepted batch has not started yet.
queued How pages will be selected.
scrape, crawl What each page will be returned as.
markdown, html Tags stored on the batch.
["docs"]
How the crawl was configured. Null for scrape batches.
Show child attributes
Show child attributes
What the submission accepted.
Show child attributes
Show child attributes
What accepting this batch cost.
Show child attributes
Show child attributes
When the batch was created.
Rejected URLs (first 100).
Show child attributes
Show child attributes
Whether this response came from cache.
Show child attributes
Show child attributes
Unique ID of this request, also in X-Request-Id. Include it when contacting support.
"3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91"
Secret for verifying X-Context-Signature. Only submit returns it, so store it.
API key usage for this request.
Show child attributes
Show child attributes
Was this page helpful?