AI Crawler Index

Erkenne KI-Crawler anhand von User-Agent oder IP und generiere die robots.txt, die deiner Haltung entspricht.

Auf workers.dev ansehenDieses MCP ausprobierenGesponsert

Dokumentation

MCP server

This index as an MCP server: Streamable HTTP at /mcp, nine read-only tools, no key, no signup. Classify a user-agent, look up a crawler, generate robots.txt, check an IP.

Connect it: one paste, one call

https://www.pathwren.workers.dev/mcp
no auth, read-only, public

That is the whole endpoint and those are its terms: Streamable HTTP (MCP), no API key, no account, no OAuth, no session to keep alive, nothing to install. Every tool is read-only, and none of them will fetch a URL on your behalf.

The JSON a connector config wants — Claude Desktop, Cursor, VS Code, Windsurf, Cline, LibreChat, Continue, anything that takes an mcpServers block. Complete as it stands; there is no field to fill in:

{
  "mcpServers": {
    "ai-crawler-index": {
      "type": "streamable-http",
      "url": "https://www.pathwren.workers.dev/mcp"
    }
  }
}

Claude Code takes one line instead:

claude mcp add --transport http ai-crawler-index https://www.pathwren.workers.dev/mcp

Then call whoami first. It takes no arguments at all, so there is nothing to invent and nothing to look up before you can see this server work — the subject of the answer is the request you just sent:

curl -s https://www.pathwren.workers.dev/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}}' \
  | jq -r '.result.content[0].text' | head -3

You are calling as: curl/8.7.1
From: 203.0.113.7 (CF-Connecting-IP)
Our own instrument books that user-agent as: agent.

Those are the answer's own first three lines. The first two are whatever YOU sent — the sample shows a plain curl from a documentation address — and the third is what this host's own classifier makes of it. The rest of the answer adds the index's own record of the user-agent you sent, whether we have seen you here before, and what our robots policy says to your user-agent. Nothing is fetched to answer it, no argument exists, and the same answer is a plain GET at /tools/whoami. example is the other zero-argument tool on this server: it runs this server's own worked example for real and hands back exactly what a real call returns. Either one is a safe first call.

The whole index, spoken as Model Context Protocol instead of as files. Streamable HTTP, stateless, no key, no signup, no session to keep alive. Every tool is read-only.

# it answers right now — no account, no header ceremony
curl -s https://www.pathwren.workers.dev/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq -r '.result.tools[].name'

Tools

ToolWhat it answers
classify_user_agentGiven a raw User-Agent header: which crawler is this, who runs it, what is it for, and what does blocking it cost me?
lookup_crawlerThe full record for one crawler by slug, name or robots.txt token.
list_crawlersThe 150 crawlers, filtered by category, operator, robots.txt stance or verification method. Returns the valid filter values too, so one call teaches the vocabulary.
generate_robots_txtA ready-to-paste robots.txt for one of 8 stances, with the reasoning and the exact tokens it implies.
is_verified_crawler_ipIs this IP inside a range the operator itself publishes? 1987 IPv4 and 1062 IPv6 prefixes, mirrored from 15 endpoints, refreshed every six hours.
whats_changedBuild time, per-source freshness, which sources changed or are failing, and the dated changelog. The cron check for anything that regenerates a config from this data.

Resources

Four, each pointing at the file the tools are computed from: agents.json, ua-regex.json, ip-ranges/all.json, status.json. Read them directly if you would rather do your own matching.

What it will not do

It has no side effects and no write path: it cannot edit your robots.txt, block anything, or fetch a URL for you. A user-agent match is a claim, not a proof — classify_user_agent says so on every answer, and is_verified_crawler_ip is the tool that turns a claim into evidence. Where an operator publishes no ranges, no tool here can verify it, and each one says which case you are in.

Measured, not asserted: what 24 hours of callers on these endpoints actually looks like — 330 client keys touched an RPC endpoint here, 214 of them never sent a POST, and 23 proved a tool call. One named window, every figure with its derivation at /data/mcp-endpoint-callers-2026-w36.json.

Measured here, not asserted: What public MCP records get wrong, measured at six endpoints — 134 client keys across 97 operators, 8,617 handshakes, and the four traps a published record walks into: the stream-open GET that answers 405, the discovery documents callers expect before they dial, two spellings of the same RFC 9728 URL, and an ownership file that 404s. One of five documents about the same 24 hours — the other four are named at the foot of each one — every one of them also at .md and .json, with the figures and the SQL under the data index beside them.

Five more servers on this host

Different jobs, no shared tool names — run all six or just the one you need. crawler-log-triage at /mcp/triage takes a whole access log and returns per-line verdicts, the impersonators, and a robots.txt or WAF ruleset for exactly what was in it. agent-discovery-doctor at /mcp/doctor turns the question around: which of the 22 discovery documents agents ask for — llms.txt, A2A agent card, owners.json, mcp.json — does a host actually serve, and who asks for each missing one. robots-policy-lint at /mcp/robots reads a robots.txt you paste and reports what it really does, by RFC 9309 rules. crawler-ip-verifier at /mcp/netcheck is the batch and prefix form of is_verified_crawler_ip: 500 addresses at a time, CIDR overlap arithmetic, and a firewall allowlist you can paste. mcp-endpoint-lint at /mcp/lint is the one aimed at MCP itself: give it another server's endpoint and it reports what the handshake negotiated, whether every advertised tool schema compiles, whether the error codes are the ones JSON-RPC requires, and what a client finds before it dials — scored out of 100.

Protocol versions 2025-06-18, negotiated per call. server/discover answers for clients on 2026-07-28, initialize for everyone else. Same data as /data/agents.json and /openapi.json — pick whichever your client speaks. Listed in the official MCP Registry as dev.workers.pathwren.www/ai-crawler-index.

Sitemap

Machine copies of this page

This document is a markdown rendering of https://www.pathwren.workers.dev/mcp.html, generated from that page's own bytes in the same build. The HTML page is canonical.