Sitemap Extractor (Apify Actor)

Follows nested sitemap indexes, reads .gz, finds sitemaps via robots.txt, returns every URL with lastmod. Bad URLs never fail the run. $0.40 per 1,000 URLs. Runs as an Apify Actor, callable from MCP clients via the Apify MCP server (https://mcp.apify.com/?actors=pennydata/sitemap-extractor-plus).

View on apify.com

Hosted MCP Server

npx add-mcp 'https://mcp.apify.com/?actors=pennydata/sitemap-extractor-plus'

Installs into Claude Code, Codex, Cursor and more

Documentation

Sitemap Extractor: Every URL from XML Sitemap to CSV/JSON (`pennydata/sitemap-extractor-plus`) Actor

Get every URL from a site's sitemap as CSV or JSON, with lastmod. Follows nested sitemap indexes, reads .gz files, finds sitemaps via robots.txt. $0.0004 per URL, bad URLs never fail the run. The prefill (docs.apify.com, 5 URLs) costs about $0.002: just click Start.

URL: https://apify.com/pennydata/sitemap-extractor-plus.md
Developed by: Penny Data (community)
Categories: SEO tools, Developer tools, MCP servers
Stats: 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
User rating: No ratings yet

Pricing

from $0.28 / 1,000 sitemap url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the API section below.

Each client library has reference documentation the quickstart does not restate: JavaScript/TypeScript (npm install apify-client) and Python (pip install apify-client).

README

Sitemap Extractor - Sitemap Index, Gzip, robots.txt, lastmod

A sitemap extractor that does not fail on messy sites: it follows nested sitemap indexes to a set depth, reads .gz sitemaps by content (even when the server mislabels them), and finds sitemaps through robots.txt. A 404 sitemap, dead host or malformed URL is logged in the SUMMARY record instead of failing the run, so you still get every URL that could be read. Same startUrls input as other sitemap extractors; no proxy setup needed.

Paste a website or sitemap URL; get one row per page URL with lastmod, priority and changefreq. $0.0004 per unique URL ($0.40 per 1,000), sitemap files, errors and duplicates are free, no login. New Apify accounts include free monthly credit, enough to try this at no cost. Click Start: the prefilled input (https://techcrunch.com, up to 25 URLs) runs as is and costs at most $0.01.

First output row (real run on https://www.sitemaps.org/sitemap.xml, 2026-10-24):

{ "url": "https://www.sitemaps.org/", "lastmod": "2016-11-21", "daysSinceModified": 3623, "status": "ok", "sitemapUrl": "https://www.sitemaps.org/sitemap.xml" }

Handles sitemap indexes, gzip, robots.txt discovery and dedupe. A dead site, a 404 sitemap or a malformed URL never fails the run: it is logged and listed in the SUMMARY record. Paid Apify plans get lower price tiers.

Quick start

Input (startUrls is the only field you need):

{ "startUrls": ["https://docs.apify.com"], "maxUrls": 20 }

Output, one row per unique URL (illustrative shape):

{ "url": "https://docs.apify.com/platform", "lastmod": "2026-09-30", "priority": 0.8, "changefreq": "weekly", "sitemapUrl": "https://docs.apify.com/sitemap.xml", "source": "https://docs.apify.com" }

Pricing event: url-extracted, $0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers); sitemap files, errors, duplicates and filtered-out URLs are free.

Example output (real run on https://www.sitemaps.org/sitemap.xml, maxUrls 20, first 5 rows):

url	lastmod	daysSinceModified	status	sitemapUrl
https://www.sitemaps.org/	2016-11-21	3623	ok	https://www.sitemaps.org/sitemap.xml
https://www.sitemaps.org/protocol.html	2022-12-15	1408	ok	https://www.sitemaps.org/sitemap.xml
https://www.sitemaps.org/faq.html	2016-11-21	3623	ok	https://www.sitemaps.org/sitemap.xml
https://www.sitemaps.org/terms.html	2016-11-21	3623	ok	https://www.sitemaps.org/sitemap.xml
https://www.sitemaps.org/da/	2016-11-21	3623	ok	https://www.sitemaps.org/sitemap.xml

Try the sample dataset: all 20 rows as JSON (public, no login). daysSinceModified is as of the run date.

What it handles

Sitemap indexes, followed recursively (configurable depth).
Gzip sitemaps (sitemap.xml.gz), detected by content, even when the server mislabels them.
robots.txt discovery: pass a home page and Sitemap: lines are found automatically; falls back to /sitemap.xml, /sitemap_index.xml, /sitemap.xml.gz.
Plain-text sitemaps (one URL per line).
Dedupe across all sitemaps and all start URLs.
Filters: include/exclude regex, modifiedSince / modifiedUntil (date or 7d), onlyChanged monitoring, maxUrls cap (also your spend cap).
Retries on network errors; 404/403 sitemaps are skipped and reported.

Input

startUrls (sites or sitemap URLs; aliases sitemapUrls, urls, sitemaps are merged with it), maxUrls (default 20), includePattern, excludePattern, modifiedSince, modifiedUntil, includeMissingLastmod, onlyChanged, maxSitemapDepth (default 5), checkPageStatus (adds an HTTP statusCode per URL via HEAD, no extra charge).

Switching from another sitemap actor? Keep your saved input and change only the Actor ID. See the Switching section below.

Drop-in for the input of apify/sitemap-extractor: startUrls as [{"url": ...}], maxCrawlingDepth (same as maxSitemapDepth) and checkPageStatus work as-is; proxyConfiguration, proxyRotation and maxRequestRetries are accepted and ignored, and no proxy is required.

Output per URL

url, lastmod, daysSinceModified, priority, changefreq, sitemapUrl (file it came from), source (your start URL). Missing fields are null.

Sitemap changed URLs: recently updated pages and lastmod filter

Use the sitemap lastmod filter to list only recently updated pages:

modifiedSince: a date (2026-10-01) or a relative age (7d, 2w, 3m, 1y). modifiedUntil sets the upper bound.
URLs with no lastmod are skipped by default (counted in skippedNoLastmod); set includeMissingLastmod: true to keep them.
Every row has a status (ok, error or skipped); unreadable sitemaps appear as free error rows with a reason. The SUMMARY record holds counts (rows, ok, error, skipped) that equal the dataset rows.
Each row gets daysSinceModified. The RUN_SUMMARY key-value record holds { total, kept, skippedNoLastmod }.

Sitemap changed URLs on a schedule (onlyChanged): the actor stores each URL's last seen lastmod in a named key-value store per site (sitemap-state-<host>). Every later run outputs only URLs that are new or whose lastmod changed, so a daily Apify schedule works as a sitemap change monitor and you pay only for changed URLs. The first run outputs all URLs as the baseline. To reset, delete that named store.

{ "startUrls": ["https://example.com"], "modifiedSince": "7d", "onlyChanged": true, "maxUrls": 1000 }

Use cases

SEO audits, content inventories, crawl seeding, change monitoring (modifiedSince, onlyChanged), site migration checklists, feeding a crawler or scraper a clean URL list.

FAQ

Does it follow a sitemap index (nested sitemaps)?

Yes. A <sitemapindex> is followed recursively down to maxSitemapDepth levels (default 5, max 10; maxCrawlingDepth is accepted as the same setting). Each sitemap file is read once. If the limit cuts off nested sitemaps, the SUMMARY record lists a "max sitemap depth reached" problem.

Can it read .xml.gz sitemaps?

Yes. A response that starts with the gzip header is decompressed whatever the URL or Content-Type says, so sitemap.xml.gz and mislabelled files both work, including gzipped files listed inside a sitemap index.

Do I need to know the sitemap URL? How does robots.txt discovery work?

No. Give a site such as https://example.com. The Actor reads its robots.txt and follows every Sitemap: line. If none is found it tries /sitemap.xml, and if that yields no URLs, /sitemap_index.xml and /sitemap.xml.gz. If you give a sitemap URL directly (ending in .xml, .txt, .gz or containing "sitemap"), robots.txt is skipped.

How do I get only URLs changed recently (lastmod filter)?

Set modifiedSince to a date (2026-10-01) or a relative age (7d, 2w, 3m, 1y); modifiedUntil sets the upper bound. URLs with no lastmod are skipped unless includeMissingLastmod is true. For a schedule, add onlyChanged to output only new or changed URLs on each run.

What does it cost?

$0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers) extracted. Sitemap files, duplicates, errors and URLs removed by your filters are free; maxUrls caps both the output and your spend.

Honest limits

Only fetches robots.txt and sitemap files that the site publishes for crawlers. It does not crawl pages, log in, or use proxies. Sites that block the request (403) are skipped.
Sites without a sitemap return nothing; this is not a link crawler.
Very large sites: raise maxUrls; each extracted URL is billed.

Switching from another sitemap actor

This Actor accepts the input field names of these Store actors, so a saved task or API call needs only the Actor ID changed. Field names were read from each actor's public input schema on 2026-10-21; output rows are this Actor's own shape (url, lastmod, priority, changefreq, sitemapUrl, source), so check any code that reads the dataset.

You used	Your input field	Here
apify/sitemap-extractor	`startUrls` (`[{"url": ...}]` or strings)	works as-is
apify/sitemap-extractor	`maxCrawlingDepth`	same as `maxSitemapDepth`
apify/sitemap-extractor	`checkPageStatus`	works as-is (adds `statusCode`, no extra charge)
apify/sitemap-extractor	`proxyConfiguration`, `proxyRotation`, `maxRequestRetries`	accepted, ignored (no proxy needed)
onescales/sitemap-url-extractor	`sitemapUrls`	works as-is (merged with `startUrls`)
pink_comic/sitemap-url-extractor	`urls`, `maxResults`, `modifiedSince`	`urls` and `modifiedSince` as-is; `maxResults` = `maxUrls`; `autoDiscover` accepted (discovery is always on); `includeImages` is not supported
mikolabs/sitemap-url-extractor	`sitemapUrls`, `maxDepth`, `filterUrlPattern`	`maxDepth` = `maxSitemapDepth`; `filterUrlPattern` = `includePattern`; `requestTimeoutSecs` accepted, ignored

Not affiliated with any of those actors. Pricing here: $0.0004 per unique URL, nothing for sitemap files, errors or duplicates.

Use from Claude, Cursor or any MCP client

Apify's MCP server can expose this actor as a tool. Use this URL (format from the Apify MCP docs, ?tools= takes a comma-separated list of actors):

https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus

Client config (mcpServers format; sign in with OAuth when the client asks):

{"mcpServers": {"apify": {
  "url": "https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus"
}}}

Example prompt: "List every URL in the sitemap of docs.apify.com, limited to 50 URLs."

Related actors

Shot PDF: capture screenshots or PDFs of the URLs you extracted.
Website Tech Stack Detector: checks the technology behind the site you mapped.
RSS/Atom Feed Reader (bulk): reads the site's feeds when it has them.

Not affiliated with sitemaps.org or any site you extract from. Written with the assistance of an AI (Claude Sonnet 5.5).

Pairs with: Website Contact Extractor

Crawl a site, then pull its contacts. The first row of each site carries a nextStep text field (untick Add next-step hint to drop it). Feed the site's homepage or its contact and about pages from this output into Website Contact Extractor: it returns the public emails, phone numbers, social links and contact-form URLs it finds on each site. It is a separate actor with its own pricing, so nothing runs or is charged unless you start it.

Actor input Schema

`startUrls` (type: `array`):

Website home pages (robots.txt is checked for Sitemap: lines, then /sitemap.xml is tried) or direct sitemap URLs (.xml or .xml.gz). Sitemap indexes are followed recursively. Aliases: sitemapUrls, urls, sitemaps (all merged).

`maxUrls` (type: `integer`):

Stop after this many unique URLs in total (and charge for at most this many). Use a higher value to extract an entire large site.

`includePattern` (type: `string`):

Only keep URLs matching this JavaScript regular expression, e.g. /blog/. An invalid regex is ignored with a warning.

`excludePattern` (type: `string`):

Drop URLs matching this JavaScript regular expression, e.g. .(pdf|jpg)$.

`modifiedSince` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this. Accepts a date (2026-01-01) or a relative age like 7d, 2w, 3m, 1y.

`modifiedUntil` (type: `string`):

Keep only URLs whose lastmod is on or before this date (YYYY-MM-DD or relative like 7d).

`includeMissingLastmod` (type: `boolean`):

When a date filter is set, also keep URLs that have no (or an unreadable) lastmod. Default false: they are skipped and counted in RUN_SUMMARY.skippedNoLastmod.

`onlyChanged` (type: `boolean`):

Remembers each URL's lastmod in a named key-value store per site (sitemap-state-) and outputs only URLs that are new or whose lastmod changed since the previous run. The first run outputs everything. Schedule it daily to monitor a sitemap for changes.

`maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow. Alias: maxCrawlingDepth.

`checkPageStatus` (type: `boolean`):

Send a HEAD request to each extracted URL and add its HTTP status code as statusCode (null if unreachable). Slower; no extra charge.

`maxCrawlingDepth` (type: `integer`):

Compatibility with apify/sitemap-extractor: same as Max sitemap index depth. Used only when maxSitemapDepth is not set.

`proxyConfiguration` (type: `object`):

Accepted for compatibility with apify/sitemap-extractor. Not required and currently ignored.

`proxyRotation` (type: `string`):

Accepted for compatibility with apify/sitemap-extractor; ignored.

`maxRequestRetries` (type: `integer`):

Accepted for compatibility with apify/sitemap-extractor; ignored (fetches use built-in retries).

`sitemapUrls` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

`urls` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

`sitemaps` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

`maxResults` (type: `integer`):

Compatibility with pink_comic/sitemap-url-extractor: same as Max URLs. Used only when maxUrls is not set.

`maxDepth` (type: `integer`):

Compatibility with mikolabs/sitemap-url-extractor: same as Max sitemap index depth. Used only when maxSitemapDepth is not set.

`filterUrlPattern` (type: `string`):

Compatibility with mikolabs/sitemap-url-extractor: same as Include regex. Used only when includePattern is not set.

`requestTimeoutSecs` (type: `integer`):

Accepted for compatibility with mikolabs/sitemap-url-extractor; ignored.

`autoDiscover` (type: `boolean`):

Accepted for compatibility with pink_comic/sitemap-url-extractor; robots.txt discovery is always on for site URLs.

`nextStepHint` (type: `boolean`):

Adds a nextStep text field on the first row of each site, pointing to a separate actor that finds contact emails and phones. Untick to leave it out.

Actor input object example

{
  "startUrls": [
    "https://techcrunch.com"
  ],
  "maxUrls": 25,
  "includeMissingLastmod": false,
  "onlyChanged": false,
  "maxSitemapDepth": 5,
  "checkPageStatus": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "nextStepHint": true
}

Actor output Schema

`results` (type: `string`):

No description

API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

JavaScript example

import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://techcrunch.com"
    ],
    "maxUrls": 25,
    "includePattern": "",
    "excludePattern": "",
    "modifiedSince": "",
    "modifiedUntil": "",
    "maxSitemapDepth": 5,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("pennydata/sitemap-extractor-plus").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Python example

from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://techcrunch.com"],
    "maxUrls": 25,
    "includePattern": "",
    "excludePattern": "",
    "modifiedSince": "",
    "modifiedUntil": "",
    "maxSitemapDepth": 5,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("pennydata/sitemap-extractor-plus").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

CLI example

echo '{
  "startUrls": [
    "https://techcrunch.com"
  ],
  "maxUrls": 25,
  "includePattern": "",
  "excludePattern": "",
  "modifiedSince": "",
  "modifiedUntil": "",
  "maxSitemapDepth": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call pennydata/sitemap-extractor-plus --silent --output-dataset

MCP server setup

{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pennydata/sitemap-extractor-plus"
        }
    }
}

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an Authorization: Bearer <APIFY_API_TOKEN> header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Coy8cbQTI06jXLXKb/builds/SbIvVa7NB2dnqhwD7/openapi.json

Sitemap Extractor (Apify Actor)

Hosted MCP Server

Documentation

Sitemap Extractor: Every URL from XML Sitemap to CSV/JSON (pennydata/sitemap-extractor-plus) Actor

Pricing

What's an Apify Actor?

How to integrate an Actor?

README

Sitemap Extractor - Sitemap Index, Gzip, robots.txt, lastmod

Quick start

What it handles

Input

Output per URL

Sitemap changed URLs: recently updated pages and lastmod filter

Use cases

FAQ

Does it follow a sitemap index (nested sitemaps)?

Can it read .xml.gz sitemaps?

Do I need to know the sitemap URL? How does robots.txt discovery work?

How do I get only URLs changed recently (lastmod filter)?

What does it cost?

Honest limits

Switching from another sitemap actor

Use from Claude, Cursor or any MCP client

Related actors

Pairs with: Website Contact Extractor

Actor input Schema

startUrls (type: array):

maxUrls (type: integer):

includePattern (type: string):

excludePattern (type: string):

modifiedSince (type: string):

modifiedUntil (type: string):

includeMissingLastmod (type: boolean):

onlyChanged (type: boolean):

maxSitemapDepth (type: integer):

checkPageStatus (type: boolean):

maxCrawlingDepth (type: integer):

proxyConfiguration (type: object):

proxyRotation (type: string):

maxRequestRetries (type: integer):

sitemapUrls (type: array):

urls (type: array):

sitemaps (type: array):

maxResults (type: integer):

maxDepth (type: integer):

filterUrlPattern (type: string):

requestTimeoutSecs (type: integer):

autoDiscover (type: boolean):

nextStepHint (type: boolean):

Actor input object example

Actor output Schema

results (type: string):

API

JavaScript example

Python example

CLI example

MCP server setup

OpenAPI specification

Sitemap Extractor: Every URL from XML Sitemap to CSV/JSON (`pennydata/sitemap-extractor-plus`) Actor

`startUrls` (type: `array`):

`maxUrls` (type: `integer`):

`includePattern` (type: `string`):

`excludePattern` (type: `string`):

`modifiedSince` (type: `string`):

`modifiedUntil` (type: `string`):

`includeMissingLastmod` (type: `boolean`):

`onlyChanged` (type: `boolean`):

`maxSitemapDepth` (type: `integer`):

`checkPageStatus` (type: `boolean`):

`maxCrawlingDepth` (type: `integer`):

`proxyConfiguration` (type: `object`):

`proxyRotation` (type: `string`):

`maxRequestRetries` (type: `integer`):

`sitemapUrls` (type: `array`):

`urls` (type: `array`):

`sitemaps` (type: `array`):

`maxResults` (type: `integer`):

`maxDepth` (type: `integer`):

`filterUrlPattern` (type: `string`):

`requestTimeoutSecs` (type: `integer`):

`autoDiscover` (type: `boolean`):

`nextStepHint` (type: `boolean`):

`results` (type: `string`):