Guides
AI Agents Overview

AI Agents Overview

SuperScraper is built to be called by AI agents as well as by humans. The REST API, the structured output format, the cost-preview flow, and the connector registry all exist so an agent can decide what to call, what it will cost, and how much to trust the result. Hosted MCP is not available.

Why agents use SuperScraper

1. REST first

Call POST /v1/scrape, /v1/extract, /v1/search, or /v1/chat with a Bearer key. Hosted MCP is not available. Local stdio MCP is developer-only (source checkout). See the MCP guide.

User: "Find 30 HVAC contractors in Phoenix with phone numbers."

Claude:
  → calls google_maps_search(query="HVAC Phoenix AZ", maxResults=30)
  → receives structured rows: name, address, phone, website, rating
  → presents a markdown table

The list_tools meta-tool lets Claude discover the full connector catalog on demand without blowing up its context budget, it asks once, gets the slugs and schemas, then calls only what it needs.

2. Structured output everywhere

Every endpoint returns a consistent envelope:

{
  "ok": true,
  "listing": {
    "name": "Apex Roofing",
    "phone": "+1 (512) 555-0100",
    "address": "1234 Main St, Austin, TX 78701",
    "website": "https://apexroofing.com",
    "rating": 4.8,
    "_confidence": 0.94,
    "_extraction_method": "json-ld",
    "_provenance": "https://www.yelp.com/biz/apex-roofing-austin"
  },
  "metadata": { "fetchMethod": "plain", "latencyMs": 412 }
}

_confidence (0–1), _extraction_method, and _provenance tell the agent how much to trust the data and where it came from, so it can decide whether to re-scrape, verify, or pass through.

3. Cost-preview before commitment

For large pulls, the agent can get a price estimate before spending budget:

POST /v1/preview
{ "connectorId": "google-maps", "query": "roofing Austin TX", "maxResults": 500 }

Response:

{
  "estimatedRows": 487,
  "estimatedCostUsd": 0.97,
  "cogsUsd": 0.31,
  "sample": [ ...first 5 rows... ],
  "jobId": "job_abc123"
}

The agent can show the user the sample and price before calling /v1/pull with the jobId to execute.

4. Chat-native entry point

POST /v1/chat accepts natural language and routes it automatically:

{ "message": "Find 50 plumbers in Austin with websites", "conversationId": "sess_xyz" }

The API:

  1. Classifies intent (lead search vs scrape vs research)
  2. Matches a curated connector or falls back to Playwright + LLM
  3. Returns a free preview (sample rows + cost)
  4. Holds the jobId for a follow-up /v1/pull

This lets an agent accept a plain English request from a user and handle the routing without any custom logic. Subsequent messages in the same conversationId can refine (kind: "refine") without re-scraping, the conversation is stateful.

5. Connectors-as-tools model

The connector registry is the catalog of things the agent can pull. Each connector has:

  • id, stable identifier (google-maps, mercado-libre, ...)
  • match(input), returns true if this connector handles the input
  • preview(params), page-1 sample + estimated cost
  • run(params), full pull, returns rows + CSV
  • schema, Zod schema of output fields (name, phone, address, website, ...)
  • pricePerRecordUsd, what the agent will pay per row
GET /v1/connectors

Returns the full catalog as JSON. Every connector is available via REST and the /v1/chat router. Hosted MCP is not available.

6. Machine-readable API discovery

  • GET /llms.txt, condensed ~150-line API reference, no auth required. Fetch once at agent session start to ground the agent on all endpoints, parameters, and pricing anchors.
  • GET /llms-full.txt, full reference with example requests and responses.
  • GET /openapi.json, OpenAPI 3.x spec for SDK generation.
import httpx
 
# Ground the agent on the API at the start of the session
api_ref = httpx.get("https://api.superscraper.dev/llms.txt").text
# Prepend to system prompt or inject as retrieval context

See the llms.txt guide for details.

7. Sync execution for agent loops

/v1/pull with a small maxResults (under ~200 rows) returns synchronously in a single HTTP response, no polling, no job IDs. The agent gets rows immediately and can continue reasoning in the same turn.

For larger pulls, the async /v1/crawl route returns a jobId and the agent polls GET /v1/jobs/:id. The response includes status: "complete" | "running" | "failed" and a progress percentage.

8. Structured error envelopes

Errors are machine-readable so the agent can act on them:

{
  "error": {
    "message": "Google Maps returned no results for this query.",
    "code": "GMAPS_NO_RESULTS",
    "suggested_fix": "Try a broader query or a different location.",
    "request_id": "gmap_lx3f8a2b"
  }
}

code is a stable enum value the agent can branch on. suggested_fix is a natural-language hint the agent can relay to the user or act on directly.

Agent patterns

Lead generation loop

1. /v1/chat  → "Find HVAC contractors in Phoenix" → preview + jobId
2. /v1/pull  → jobId → rows (name, phone, address, website)
3. /v1/batch → scrape each website → listing data (email, tech stack)
4. /v1/enrich/website → enrich top 10 by rating → team contacts
5. Present CSV to user

Research + enrich

1. /v1/enrich/website → emails, phones, tech stack, team from a domain
2. /v1/extract with schema → typed fields with _confidence + _provenance

Competitive intelligence

1. /v1/map → enumerate all pages on a competitor's site
2. /v1/batch → scrape pricing/features/blog pages
3. /v1/extract --schema → pull structured pricing table
4. Compare against your own product

Monitor for new listings

POST /v1/feeds
{
  "connectorId": "google-maps",
  "params": { "query": "new roofing contractors Austin TX" },
  "schedule": "0 9 * * 1",   // every Monday at 9am
  "webhookUrl": "https://your-app.com/webhook"
}

The feed worker re-runs the pull on schedule, deduplicates against the last run, and delivers only new rows to the webhook. The agent subscribes once and receives a stream of fresh leads.

Quick reference, agent-friendly endpoints

EndpointSync?Returns
POST /v1/chatYesIntent classification + preview + jobId
POST /v1/previewYesCost estimate + sample rows + jobId
POST /v1/pullYes (small)Structured rows + CSV
POST /v1/scrapeYesMarkdown + listing JSON
POST /v1/extractYesArbitrary JSON matching your schema
POST /v1/batchYesUp to 100 URLs concurrently
POST /v1/crawlNojobId → poll /v1/jobs/:id
GET /v1/connectorsYesConnector catalog as JSON
POST /v1/feedsYesCreates a scheduled monitor
GET /llms.txtYesPlain-text API reference (no auth)

All endpoints accept Content-Type: application/json and authenticate via x-api-key header.

Getting started

  1. Get a free API key at app.superscraper.dev (opens in a new tab).
  2. Call the REST API directly from your agent, see the API Reference. Hosted MCP is not available.
  3. Local stdio MCP is developer-only; see the MCP guide.
  4. Optional Claude Code plugin from a source checkout, see the Plugin guide.
  5. Fetch /llms.txt to ground your agent on the full API surface.
PricingGET /v1/jobs