POST /v1/map

Return all URLs found on a website, synchronously, in a single request. Useful for discovering pages before a targeted crawl or scrape.

The engine tries sitemap.xml and sitemap_index.xml first. If no sitemap exists, it falls back to parsing internal href links on the homepage.

Request

POST /v1/map
Content-Type: application/json
x-api-key: YOUR_KEY

Body

Field	Type	Required	Default	Description
`url`	string	Yes		The website to map. Can be the homepage or any page on the domain.
`limit`	number	No	`100`	Maximum URLs. Clamped by plan (free max 100 map links).
`search`	string	No		Filter results to URLs containing this string (case-insensitive).

Example, map a full site

curl -X POST https://api.superscraper.dev/v1/map \
  -H "x-api-key: ss_live_xxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.acmeplumbing.com",
    "limit": 200
  }'

Example, find all service pages

curl -X POST https://api.superscraper.dev/v1/map \
  -H "x-api-key: ss_live_xxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.acmeplumbing.com",
    "search": "/services/"
  }'

Response

HTTP 200 when the site was reachable (including zero links).
HTTP 502 map_fetch_failed when fetch failed and no links were found.
HTTP 504 map_timeout when the time budget expired with no links.

Field	Type	Description
`url`	string	Normalized base URL
`links`	string[]	Deduplicated list of discovered URLs
`total`	number	Number of URLs in `links`
`source`	string	Where links came from: `"sitemap"`, `"crawl"`, or `"none"`
`timedOut`	boolean \| undefined	Present when the 25s budget was hit but some links may still be returned

Example response

{
  "url": "https://www.acmeplumbing.com",
  "links": [
    "https://www.acmeplumbing.com/",
    "https://www.acmeplumbing.com/services",
    "https://www.acmeplumbing.com/services/drain-cleaning",
    "https://www.acmeplumbing.com/services/water-heaters",
    "https://www.acmeplumbing.com/services/sewer-repair",
    "https://www.acmeplumbing.com/about",
    "https://www.acmeplumbing.com/contact",
    "https://www.acmeplumbing.com/reviews",
    "https://www.acmeplumbing.com/service-areas/austin",
    "https://www.acmeplumbing.com/service-areas/round-rock",
    "https://www.acmeplumbing.com/service-areas/cedar-park",
    "https://www.acmeplumbing.com/blog",
    "https://www.acmeplumbing.com/blog/how-to-unclog-a-drain"
  ],
  "total": 13,
  "source": "sitemap"
}

Source values

Value	Meaning
`"sitemap"`	Links were parsed from `sitemap.xml` or `sitemap_index.xml`
`"crawl"`	Sitemap not found, links extracted from `href` attributes on the homepage
`"none"`	No links could be discovered

Sitemap index support

When a site has a sitemap_index.xml, the engine automatically fetches up to 15 child sitemaps and merges all <loc> entries before applying the limit and search filter. Discovery runs under a 25-second budget; if it is exceeded the response returns whatever links were found so far with timedOut: true.

Crawl Batch