POST /v1/crawl

Crawl an entire website asynchronously, following internal links up to a configurable page limit. The crawl is enqueued as a background job and returns a jobId immediately. Poll GET /v1/jobs/:id to check status and retrieve results.

Optionally provide a webhookUrl to receive a POST callback when the crawl completes.

Start a crawl

POST /v1/crawl
Content-Type: application/json
x-api-key: YOUR_KEY

Body

Field	Type	Required	Default	Description
`url`	string	Yes		Starting URL. The crawl stays on the same domain.
`maxPages`	number	No	`10`	Maximum pages to scrape. Clamped to 1000 and to your plan quota.
`limit`	number	No		Alias for `maxPages`. Takes precedence when both are set.
`maxDepth`	number	No		Maximum link depth from the seed URL (seed = depth 0).
`includePaths`	string[]	No		Only crawl URLs whose path matches one of these prefixes/globs.
`excludePaths`	string[]	No		Skip URLs whose path matches one of these prefixes/globs.
`allowBackwardLinks`	boolean	No	`false`	Allow following links that climb above the seed path.
`ignoreRobotsTxt`	boolean	No	`false`	Respect robots.txt by default; set `true` to opt out (Firecrawl-compatible).
`webhook`	string	No		Completion webhook URL. `webhookUrl` is accepted as an alias.

Async crawl needs a job queue and a job store. If REDIS_URL or the database is not configured, the call returns 503 with status: "failed" instead of queuing a job that would never run.

Example request

curl -X POST https://api.superscraper.dev/v1/crawl \
  -H "x-api-key: ss_live_xxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.acmeplumbing.com",
    "maxPages": 25,
    "webhookUrl": "https://myapp.example.com/webhooks/crawl-done"
  }'

Response (HTTP 202)

{
  "jobId": "01hwz3p4q8k2nmx7vyd6cbse1f",
  "status": "queued",
  "url": "https://www.acmeplumbing.com",
  "message": "Crawl queued. Poll GET /v1/jobs/:id for status."
}

GET /v1/jobs/:id

Poll a crawl job for its current status and results.

GET /v1/jobs/01hwz3p4q8k2nmx7vyd6cbse1f
x-api-key: YOUR_KEY

Response fields

Field	Type	Description
`id`	string	Job ID
`url`	string	Starting URL
`status`	string	`"queued"`, `"running"`, `"processing"`, `"completed"`, `"failed"`, or `"cancelled"`
`result_url`	string \| null	Signed URL to download the full results JSON (present when `status: "completed"`)
`pages_scraped`	number	Pages scraped so far
`error`	string \| null	Error message (present when `status: "failed"`)
`created_at`	string	ISO 8601 timestamp
`completed_at`	string \| null	ISO 8601 timestamp when the job finished

Example, job in progress

{
  "id": "01hwz3p4q8k2nmx7vyd6cbse1f",
  "url": "https://www.acmeplumbing.com",
  "status": "running",
  "result_url": null,
  "pages_scraped": 8,
  "error": null,
  "created_at": "2026-05-30T14:22:10Z",
  "completed_at": null
}

Example, job done

{
  "id": "01hwz3p4q8k2nmx7vyd6cbse1f",
  "url": "https://www.acmeplumbing.com",
  "status": "completed",
  "result_url": "https://storage.superscraper.dev/results/01hwz3p4q8k2nmx7vyd6cbse1f.json?token=...",
  "pages_scraped": 24,
  "error": null,
  "created_at": "2026-05-30T14:22:10Z",
  "completed_at": "2026-05-30T14:24:37Z"
}

GET /v1/jobs

List your 20 most recent crawl jobs.

GET /v1/jobs
x-api-key: YOUR_KEY

Response

{
  "jobs": [
    {
      "id": "01hwz3p4q8k2nmx7vyd6cbse1f",
      "url": "https://www.acmeplumbing.com",
      "status": "completed",
      "pages_scraped": 24,
      "created_at": "2026-05-30T14:22:10Z",
      "completed_at": "2026-05-30T14:24:37Z"
    },
    {
      "id": "01hwz2r9j5m1klx4uac3byqd8e",
      "url": "https://www.roofingpros.com",
      "status": "failed",
      "pages_scraped": 3,
      "created_at": "2026-05-29T09:15:44Z",
      "completed_at": "2026-05-29T09:16:02Z"
    }
  ]
}

GET /v1/crawl/:id

Firecrawl-parity alias for GET /v1/jobs/:id, scoped to your tenant. Returns the same job status fields.

GET /v1/crawl/01hwz3p4q8k2nmx7vyd6cbse1f
x-api-key: YOUR_KEY

DELETE /v1/crawl/:id

Cancel a running crawl. Marks the job cancelled so the worker stops persisting new pages and you stop polling. A job that has already reached a terminal state (completed, failed, or cancelled) reports that state instead of being cancelled. Tenant-scoped and idempotent.

curl -X DELETE https://api.superscraper.dev/v1/crawl/01hwz3p4q8k2nmx7vyd6cbse1f \
  -H "x-api-key: ss_live_xxxxxxxxxxxxxxxxxxxx"

{ "jobId": "01hwz3p4q8k2nmx7vyd6cbse1f", "status": "cancelled", "cancelled": true }

Webhook payload

When a webhookUrl is provided, SuperScraper POSTs to it with the completed job payload:

{
  "jobId": "01hwz3p4q8k2nmx7vyd6cbse1f",
  "status": "completed",
  "url": "https://www.acmeplumbing.com",
  "pages_scraped": 24,
  "result_url": "https://storage.superscraper.dev/results/01hwz3p4q8k2nmx7vyd6cbse1f.json?token=...",
  "completed_at": "2026-05-30T14:24:37Z"
}

Respond with any 2xx status to acknowledge the webhook.

Search Map