AI Crawler Access Check by Citant.ai

Checks whether ChatGPT, Claude, Perplexity and other AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) can actually reach a website. Fetches the site as each crawler, so it catches Cloudflare, Akamai and other firewall/CDN blocks that a robots.txt reader misses. Returns a 0-100 AI Access Score, per-crawler verdicts and fixes. Free, read-only, no login.

View on citant.ai

Hosted MCP Server

npx add-mcp 'https://check.citant.ai/mcp?src=mcpservers'

Installs into Claude Code, Codex, Cursor and more

Documentation

AI Crawler Access Check MCP server

A free, remote MCP server that lets your AI app answer one question: can ChatGPT, Claude, Perplexity and the other AI crawlers actually reach a website? It runs the same 21 checks as the AI Crawler Access Report, fetching the site as each crawler, so it catches firewall and CDN blocks that a robots.txt reader misses. Read-only, no login, no API key.

Server URL: https://check.citant.ai/mcp
Transport: Streamable HTTP. Tool: check_ai_crawler_access (input: a domain or URL).

Add it to your AI app

Claude (web and desktop)

Open Settings, then Connectors, choose Add custom connector, give it a name and paste the URL. No authentication.

https://check.citant.ai/mcp?src=docs

Claude Code

Run once in a terminal:

claude mcp add --transport http citant-crawler-check "https://check.citant.ai/mcp?src=docs"

Cursor

Add to ~/.cursor/mcp.json (or.cursor/mcp.json in a project):

{
  "mcpServers": {
    "citant-crawler-check": {
      "url": "https://check.citant.ai/mcp?src=docs"
    }
  }
}

VS Code

Add to.vscode/mcp.json:

{
  "servers": {
    "citant-crawler-check": {
      "type": "http",
      "url": "https://check.citant.ai/mcp?src=docs"
    }
  }
}

ChatGPT (developer mode) and other MCP apps

Create a custom connector that points at this URL, with no authentication:

https://check.citant.ai/mcp?src=docs

What to ask

  • Can ChatGPT and Perplexity crawl example.com?
  • Is my site blocking GPTBot at the firewall, or only in robots.txt?
  • Check AI crawler access for our pricing page and tell me the top fix.

What it returns

  • An AI Access Score from 0 to 100 with three sub-scores (live answers, training data, search index)
  • A verdict for each crawler: what robots.txt says and what a real fetch as that crawler got
  • Up to 8 critical and warning findings, each with a fix
  • A link to the full report on check.citant.ai

A scan takes 15 to 60 seconds. Results for the same domain are cached for 24 hours. The tool reads public pages only and never changes anything. What it fetches and stores is documented on the CitantScanBot page.

Where it is listed

Prefer a browser?

Run the same check on check.citant.ai. If the scan shows you are readable but AI assistants still do not name you, that is a different problem, and it is what the AI Search Audit by Citant.ai measures.