Doormat CrawlerCheck
通过 MCP 检查 AI 爬虫访问权限,免费。工具 check_ai_crawler_access 会报告你的 robots.txt 允许或屏蔽了哪些具名 AI 机器人(GPTBot、ClaudeBot 等)。付费的 /report(0.03 USDC,通过 Base 上的 x402)会提供通俗易懂的建议和可直接粘贴的 robots.txt 代码片段。
文档
crawlercheck
Checks which of the ~20 named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended and more) your site's robots.txt currently allows or blocks.
Free, no signup. Paid (0.03 USDC on Base): /report?url= adds a plain-English recommendation and a ready-to-paste robots.txt snippet.
crawlercheck by Doormat Capital — free AI-crawler robots.txt checker. Submitted to Twelve Tools.
Our fortnightly notes on what we ship: doormat.sebastiaan-ba3.workers.dev