Agents
llms.txt

llms.txt, AI Agent Discovery

SuperScraper exposes two machine-readable discovery files that follow the llms.txt standard (opens in a new tab). These are plain-text files designed to give AI coding assistants and autonomous agents a concise, grounded understanding of the API without requiring them to crawl HTML docs.

Both files are public, no API key is required.

Endpoints

PathContentTypical use
GET /llms.txt~150-line condensed reference: base URL, auth, all endpoints with key params, pricing anchors, response envelopeInitial orientation, context window budget is tight
GET /llms-full.txtFull reference: every endpoint with example request and response bodiesDeep integration, writing SDK wrappers, code generation

What's in the files

/llms.txt (condensed)

  • Product description and base URL
  • Authentication instructions (x-api-key: ss_<key>)
  • Every endpoint group (core scraping, scrapers, enrichment, KYB, billing) with key parameters
  • Pricing anchors (plan names and monthly scrape/extraction quotas)
  • Response envelope description (success / error shape, _confidence, _extraction_method, _provenance)
  • Pointer to /llms-full.txt

/llms-full.txt (full)

Everything in /llms.txt plus:

  • Full request and response JSON examples for every endpoint
  • Error shape examples (400, 401, 422, 429, 500)
  • Rate limit table
  • SDK / OpenAPI references

Using with AI agents

Fetch /llms.txt at the start of an agent session to ground it on the API:

import httpx
 
# No auth required
ref = httpx.get("https://api.superscraper.dev/llms.txt").text
# Prepend to system prompt or use as retrieval context
const ref = await fetch("https://api.superscraper.dev/llms.txt").then(r => r.text());
// Inject into agent system prompt

For Claude Code / Claude Desktop via the MCP server, the tool descriptions already embed the key endpoint signatures, you do not need to fetch llms.txt manually.

For Claude Desktop (llms.txt plugin)

If you use a browser plugin or MCP tool that auto-fetches llms.txt files, add:

https://api.superscraper.dev/llms.txt

to your plugin's site list. The file includes Full index: /llms-full.txt so the plugin can follow through to the full reference.

Caching

Both files are served with Cache-Control: public, max-age=3600. The files are updated on each API release, versioned API changes will be reflected within an hour.

Format notes

  • Both files are text/plain; charset=utf-8
  • No authentication headers are needed or accepted for these paths
  • The files are mounted at the root level (not under /v1/) and are exempt from the requireApiKey middleware
SKILL.md onboardingCLI