CrawlCheck

Checks a website before an AI agent reads, cites, connects to or pays it. Returns a signed allow/warn/block decision, a graded record of what AI crawlers receive, robots.txt and llms.txt readings, and a site-safety check.

Máy chủ MCP được lưu trữ

npx add-mcp 'https://crawlcheck.io/mcp'

Cài vào Claude Code, Codex, Cursor và nhiều công cụ khác

Tài liệu

Free check for your website

Can AI read your site?

See what your website serves to AI crawler request profiles. CrawlCheck compares real responses from its own network, names what went wrong, and shows you what to fix first.

Domain

Check my site

First scan free with your email, then 3 credits ($3) a scan. Nothing installed, nothing changed on your site. 52 sections, read as 16 named crawler request profiles and 3 comparison controls. Example report · a public site, checked daily www.denverpost.com Score 81 / 100 8 things to fix Share of the page text each crawler profile received GPTBot 100% ClaudeBot 100% Googlebot 100% Bytespider 403 AI assistants like ChatGPT can read this site, but find little they can quote from its pages. See the full report →

GPTBot ClaudeBot PerplexityBot Googlebot

scans to Denver

Where scans come from. Every dot is a city or country a visitor has run a scan from, shown at city level. Open the globe

551,295sites measured

8,418scans run

52sections in every audit

16 + 3crawler profiles and controls

104findings published

READ AS GPTBot ClaudeBot OAI-SearchBot Claude-SearchBot PerplexityBot ChatGPT-User Claude-User Perplexity-User Googlebot Googlebot Smartphone Bingbot Applebot Amazonbot Bytespider Meta-ExternalAgent CCBot + controls

01 · How it works

See exactly what each request received.

We compare what your site serves to search crawlers, training crawlers, user-triggered fetchers and controls from CrawlCheck's network. That includes separate desktop and smartphone Googlebot requests. We also read your robots.txt, machine files and declared agent permissions. The answer is graded by published rules and signed, so anyone can check it offline.

The full definition

CrawlCheck is the verification layer for the agentic web. It measures what a site actually serves — tested with named crawler request profiles, including desktop and smartphone Googlebot, alongside its machine files, crawl policy, declared rights and capabilities — decides findings by published rules, and signs the answer so an agent can verify it offline before it reads, cites, connects or acts. Site owners get the same measurement as a free A–F audit with every defect named.

01

We ask as each crawler

One address, the same page, requested as each of the 16 named crawler profiles plus a desktop browser, a smartphone Googlebot and plain controls.

GPTBot ClaudeBot PerplexityBot Googlebot + 12 more

02

We compare what came back

Status, size, content and cache state for each of the 19 replies, side by side, with your robots.txt, machine files and declared agent permissions read alongside.

Browser control GPTBot same ClaudeBot different PerplexityBot refused Illustration, not a real scan

03

We name what to fix first

A grade from A to F across 52 sections, every finding shown in full with its fix steps, and the one thing to fix first at the top.

A B C D F

Paying never changes a reading. The scan, step by step

02 · Independent research

Being readable beats being talked about.

In 37,927 AI agent sessions about 1,056 real businesses, agents did better on every measure when they could read the business’s own site.

Source: Finder, Elovic, Shalev and Yosef, “AX is the New AEO”, arXiv:2609.34951, 29 September 2026. CrawlCheck was not involved in the study.

**1.9×**as often clearly recommended

**78%**of answers built from the business’s own pages, against 56%

**41%**more accurate answers

**7–10%**of the answer came from the model’s memory

03 · Four doors

Who it is for.

01 AgentsOne signed yes-or-no before your agent acts on a site. Quickstart · Resolve · SDKs 02 Site ownersCheck free. We fix it for $100. Watch re-checks twice a day and alerts you if it regresses, from $29 a month. Scan · Fix · Watch 03 Developers and vendorsThe API, typed SDKs, the specs and the signing keys. API · Specs · Keys 04 Researchers and pressA daily census of the top 1,000 sites, published as open CC BY 4.0 data that anyone can check. Census · Conduct · Datasets

04 · Questions

What people ask before they check.

Can ChatGPT read my website?

ChatGPT reaches the web through more than one crawler: OAI-SearchBot finds pages for ChatGPT search, and GPTBot collects content for model training. Allowing GPTBot is not needed to appear in ChatGPT search, and blocking it does not remove you. Many sites block one or both in robots.txt or behind a bot wall without knowing. That is the access we can test, not the only way an AI service may reach your content. The free check fetches your homepage as OAI-SearchBot, GPTBot and fourteen other crawlers from our own network and shows exactly what each one got.

Why doesn’t my business show up in AI answers?

Three usual reasons: the crawler is blocked, the page it gets has no readable text, or there is no clear fact to quote: a name, address, hours and prices that match everywhere. The report names which one applies to you.

Is the check free, and does it change anything on my site?

Free, no account. Every request is a read, the same GET a crawler makes. Nothing is installed, submitted or changed.

What does registering my site do?

It tells every agent that asks who runs the site and what it declares, verified by a fetch. Free, from the registry page.

Videos in production

In production Start here ### Case studiesIn production Shorts ### Can AI read your website? One-minute checksIn production How-to ### Get recommended by AI: fixes for local businesses

Subscribe on YouTube→

Featured articles

Finding ### State of AI visibility, October 2026: 139,788 homepages 139,788 homepages measured 3–6 October 2026: 6.3% block an AI crawler, 15.4% publish llms.txt, the median homepage is 3.6% text. Live defect rates inside. 2026-11-02 · 7 min read Read →Finding ### Google's Googlebot check passes 198 of 200 rented servers Google's three-domain Googlebot rule accepts any Google Cloud server: 198 of 200 sampled addresses passed. Two tools tested, the fix, and how to reproduce. 2026-11-02 · 6 min read Read →Finding ### AI picked a local contractor 422 times: rating decided it 422 paired tests: dropping a rating from 4.8 to 4.1 stars pushed a business down an AI ranking every time. Review count and area served came next. 2026-11-02 · 3 min read Read →

Browse the full library→

Machine layer llms.txt↗llms-full.txt↗agents.md↗SKILL.md↗entitymap.json↗media kit↗MCP server card↗API catalog↗OpenAPI↗robots.txt↗sitemap.xml↗security.txt↗

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. Who runs this · Press kit · Videos · Evidence for disputes · YouTube · hello@crawlcheck.io

How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. What we collect and how to opt out · how every report is timestamped · who runs this · press kit · hello@crawlcheck.io.

Second vantage: the browser extension measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

Our own machine layer for the agentic web: llms.txt · llms-full.txt · agents.md · SKILL.md · entitymap.json · media kit · MCP server card · API catalog · OpenAPI · robots.txt · sitemap.xml · security.txt — every one named in robots.txt, which is exactly what we check for on yours. Home page updated 2026-11-17 .

Pages: Keys · Listing check · SDK · Certificates | Glossary: DataDownload | Directory: Vespa | Registry: bobsteelechevy.com · deobeautyusa.com · griptape.ai · machsoft.com.tr · plombier-paris.fr · think.resoneo.com

Contact · Partnerships · Terms of service · Refund policy · Trust center · Subprocessors · Changelog · Report an error