Site SEO & AI Crawler Audit

npx add-mcp 'https://sitecheck.openkrill.app/mcp'

Questions it answers

Site SEO and AI Crawler Audit checks public web pages and reports what they publish right now: SEO tags, redirects, AI crawler rules, AI citation signals, landing page conversion and, for a site you own, what leaks publicly. People also call it Site Check.

The exposure check works only on a domain you prove you own: the first call gives a DNS TXT record or a file to publish, and the check runs once it is live. It never looks up a person, a user name or an email address.

Page and crawler checks: which of 14 AI crawlers robots.txt allows on a page and the rule that decided it; the title, description, canonical, Open Graph and Twitter tags and which are missing; and the 301 redirect chain with status codes and headers.

Store and AI checks: one product page's SEO and Product structured data (price, GTIN, availability); a domain's llms.txt, agents.md, UCP file, sitemap and AI crawler rules; five AI citation signals of one page, each with evidence and a fix; and a sample of up to 40 sitemap addresses for broken pages and redirects.

Landing page check: twelve conversion items such as the h1, the call to action and whether its links work, form fields, trust signals, a price section, viewport and a contact path, each with evidence and a fix. It is a checklist, not a forecast.

Exposure check: exposed or clean with the top 3 risks and fixes, from dangling subdomains, leaked files such as.env, admin paths and masked API keys in the page's scripts, plus six legal signals with official sources. No port scan, and not legal advice.

AI-built app check: for an app made with Lovable, Bolt, v0, Replit or any AI app builder, the missing security headers of any public app with the exact header lines for its host; and, for an app you prove you own, exposed files such as.env and.git, keys in the JavaScript it ships, and whether its Supabase tables or Firebase database, Firestore and storage rules let anyone read them. It reads only status codes and Supabase row counts, never a row's data, and never writes.

What it does not do: no speed test, no rankings or traffic, no JavaScript, no sign-in, no page text returned, and no private, local or internal addresses. It does not request a page that robots.txt disallows for Agent Tools. robots.txt is a request, not access control, and a firewall can treat real crawlers differently.

It stores nothing about you or the site. Answers are cached for five minutes, except owner checks (exposure_check, ai_app_check), which are never cached; the only lasting record is a daily count of calls per tool.

What it can do

• Shows whether robots.txt allows or blocks 14 published AI crawlers
• Shows the rule that decided each crawler, the sitemaps and any noindex or noai directives
• Reads a page's title, description, canonical link, language and icons
• Reads Open Graph and Twitter share tags and lists the common ones that are missing
• Follows the redirect chain with each status code and shows the final address and headers
• Checks one product page's title, description, h1, canonical link and Product structured data (price, GTIN, availability)
• Checks a domain for llms.txt, agents.md, the Shopify UCP file, a sitemap and AI crawler rules in robots.txt
• Checks a page for five AI citation signals: summary, author credentials, Q&A format, headings, structured data
• Checks up to 40 sitemap addresses for broken pages and redirect chains, skipping pages robots.txt disallows
• Checks a landing page's h1, call to action, form fields, trust signals, listed prices, viewport and contact path
• Tests up to five call-to-action links of a page and gives each finding with evidence and a fix
• Checks the exposed surface of a domain its owner proves: leaked files, dangling subdomains, default pages
• Finds API keys in the JavaScript an owned app ships, public source maps and open admin paths, keys masked
• Flags 6 legal signals on an owned app (COPPA, fonts, session replay, CAN-SPAM, renewals, DMCA) with sources
• Checks 8 security headers of any public app and gives the exact lines for Vercel, Netlify, Cloudflare and others
• Tells an owner whether anyone can read their Supabase tables or Firebase data, by status and row count only
• Refuses private, local and internal addresses and redirects into them

Tools

• check_ai_crawler_access, Check AI crawler access in robots.txt. Checks whether 14 AI crawlers may fetch a page. Use it when the user asks whether AI crawlers can read a website or page, for example "can AI crawlers read example.com?", "is GPTBot blocked in my robots.txt?" or "which AI bots are allowed on https://example.com/blog?". Pass the page address. Returns, for 14 AI crawlers, whether robots.txt allows the page and the rule that decided it, plus the sitemaps, the redirect chain, and the page's X-Robots-Tag header and robots meta tag. It reads robots.txt only and does not test whether a firewall blocks crawlers. Do not use it for private or local addresses, for security scans, or to collect a site's content.
• check_page_tags, Check Open Graph and share tags. Reads title, description and share tags of a page. Use it when the user asks about the meta tags, canonical tag or Open Graph social preview of a web page, for example "what will the preview look like when I share this page?", "does my homepage have a meta description and canonical tag?" or "which Open Graph tags is my page missing?". Pass the page address. Returns the title, meta description, canonical link, language, viewport, icons, Open Graph and Twitter tags, and a list of the common tags that are missing. Tags added by JavaScript are not seen. Do not use it to read a page's text or contact details, for private or local addresses, or for rankings and traffic.
• trace_redirects, Trace a 301 redirect chain. Traces the redirects and headers of a web address. Use it when the user asks where a web address redirects to, or for a 301 redirect chain, for example "why does http://example.com end up on another address?", "show the redirect chain for this link" or "does this site redirect http to https?". Pass the address. Returns each hop with its status code and target, the final address and status, whether http is upgraded to https, and the main response headers such as cache-control and strict-transport-security. At most 6 redirects are followed. Do not use it for private or local addresses, to read page content, or to scan a site for vulnerabilities.
• product_page_seo, Check product JSON-LD and page SEO. Checks the SEO and Product data of a product page. Use it when the user asks whether one product page of an online store is set up for search, for example "is https://shop.example.com/products/blue-mug set up for search?", "does my product page have Product structured data?" or "why does Google show no price for this product?". Pass the product page address. Returns title and meta description length, h1 headings, canonical link, Open Graph tags, and the Product structured data (JSON-LD): name, brand, SKU, GTIN, MPN, price, currency, availability, variant count and rating summary, plus a list of issues found. On Shopify it adds the product handle and whether a variant parameter is in the address. Works on any store. It reads one page, skips it if robots.txt disallows it for Agent Tools, and does not run JavaScript. Do not use it for page speed, rankings or traffic, to collect prices or catalogs, or for private or local addresses.
• ai_readiness_check, Check AI readiness of a store or page. Checks if AI agents can read and cite a store or page. Use it when the user asks whether AI agents can read an online store or cite a page, for example "does shop.example.com have an llms.txt?" or "will AI assistants cite this page?". Pass url for a page or domain for the domain files only; one is enough. Returns whether the domain has /llms.txt, /agents.md, the UCP file at /.well-known/ucp (version and services shown) and /sitemap.xml, and which of 14 AI crawlers robots.txt blocks site-wide. With a page address it also rates five citation signals (summary near the top, author credentials, Q&A format, heading structure, structured data) as pass, warn or fail with evidence and a fix. It requests at most five domain files and one page, skips what robots.txt disallows for Agent Tools, and shows what is published, not whether an agent can buy. Do not use it for private or local addresses, to read page content, or to scan for vulnerabilities.
• link_sample_check, Check a sitemap sample for broken links. Finds broken or redirected pages in a sitemap sample. Use it when the user asks whether pages of an online store or website are broken or redirected, or for a broken link in the sitemap, for example "are any pages of shop.example.com broken?", "check my sitemap for 404s" or "do my product pages redirect?". Pass the domain. Reads the sitemap (for a sitemap index, the product, page, collection and blog sitemaps first) and checks up to 40 addresses spread across it, on that host only, returning counts by status, and for each broken, unreachable or redirected address its status and final address. It is a sample, not every page; it uses HEAD requests at low concurrency, skips addresses that robots.txt disallows for Agent Tools, and makes at most 46 requests. Do not use it for private or local addresses, to read page content, or to monitor a site over time.
• landing_page_check, Check a landing page for conversion. Checks a landing page for conversion problems. Use it when the user asks how to get more sign-ups, sales or enquiries from a landing page, for example "review https://example.com/ for conversion" or "does my landing page have a clear call to action?". Pass the page address. Returns twelve checks, each with a status, evidence and a fix: the h1 and the sentence under it, the primary call to action and its repeats, whether up to five call-to-action links work, form field count, trust signals, a visible price or next step, the viewport tag, HTML size, render-blocking hints and a contact path. It reads one page, tests a few links, skips what robots.txt disallows for Agent Tools, does not run JavaScript, and returns no page copy, reviews or contact details. It is a checklist, not a conversion forecast, a speed test or a design review. Do not use it for private or local addresses, rankings or traffic.
• exposure_check, Check the exposed surface of a site you own. Checks what a site or app the user owns leaks publicly. Use it when the user asks what an attacker could find on their own domain, host or app, for example "does my app leak a.env file?". Pass the domain, host or app page address, not a bare IP. The first call returns ownership_not_proven with a DNS TXT record or file: publish it, then call again with the same target. Then it gives exposed or clean, the top three risks with fixes, and evidence: dangling subdomains, leaked files (never their contents), masked API keys, source maps, admin paths, dev servers and default pages. It adds six legal signals with law and source (COPPA, Google Fonts, session replay, renewals, DMCA, CAN-SPAM for a passed email); not legal advice. Findings are labeled confirmed, needs verification or rejected. GET only, at most 32 fixed paths; no crawl, sign-in or port scan. Do not use it on a target the user does not own, to look up a person, user name or email, or for private or local addresses.
• ai_app_check, Check an AI-built app for leaks and open data. Checks an app made with an AI app builder (Lovable, Bolt, v0, Replit, Cursor) for leaks and launch readiness. Use it when the user asks "is my Lovable app secure?", "is my Supabase data public?", "which security headers is my app missing?" or "is my app ready to launch?". Pass the app address. For any public app it returns 8 security headers with the exact lines for the detected host (Vercel, Netlify, nginx and more), and launchList: 20 items of a 160-item launch list read from outside (renewals, mail records, security.txt, caching, legal links), each pass, fail or warn with a fix; checklist true adds the owner-only items with how to check each. The first call also returns ownership_not_proven with a DNS TXT record or file: once the owner publishes it, the next call adds exposed files (.env,.git), masked keys in the shipped JavaScript, source maps, open Supabase or Firebase data, and backend signals. GET only; database checks read status codes and row counts, never a row's data, and never write. Do not use it on an app the user does not own beyond the public checks, for private or local addresses, or to scan many sites.

Add Site SEO & AI Crawler Audit in Claude

Claude connects to a remote MCP server as a custom connector. The connection is made from Anthropic's cloud, not from your laptop, so the server must be on the public internet. Site SEO & AI Crawler Audit is. These steps were checked against Claude's custom-connector article (updated 2026-08-11) and the Claude Code MCP docs on 2026-10-01.

On a Pro or Max plan:

1. Open Customize, then Connectors.
2. Select the plus button, then Add custom connector.
3. Paste https://sitecheck.openkrill.app/mcp. Leave Advanced settings empty: this server does not need an OAuth client id or secret.
4. Select Add. The connector is yours. Anthropic has not reviewed it, and it is not a Directory listing.
5. In a conversation, open the plus button, then Connectors, and enable Site SEO & AI Crawler Audit. Ask: Will AI assistants cite https://example.com/blog/guide?

On Team and Enterprise, an Owner adds the connector first at Organization settings, then Connectors, then Add, then Custom, then Web, and pastes the same URL. Members then connect it from Customize. A Free plan can add one custom connector.

In Claude Code, add it from the project directory:

claude mcp add --transport http site-check https://sitecheck.openkrill.app/mcp

That writes a local or user config. A JSON entry needs "type": "http" (or streamable-http) plus the url. An entry with a url and no type is skipped. Confirm with claude mcp get site-check. The tools you should see are check_ai_crawler_access, check_page_tags, trace_redirects, product_page_seo, ai_readiness_check, link_sample_check, landing_page_check, exposure_check, ai_app_check.

If Claude tries to fetch https://sitecheck.openkrill.app/mcp from its code sandbox and reports HTTP 403, the block is Claude's sandbox network allowlist, not this server. The sandbox proxy answers 403 with x-deny-reason: host_not_allowed for every host outside the allowlist. Use the connector above instead: connector calls come from Anthropic's cloud and do not pass the sandbox proxy. To let the sandbox reach it as well, an Owner adds sitecheck.openkrill.app under Settings, then Capabilities, then Additional allowed domains.

Enable the connector only in conversations that need it. Claude can call the tools while you work, including during research, so turn off any tool you do not want invoked. Site SEO & AI Crawler Audit does not write to your accounts. It still receives the arguments of each call. Read the privacy page before you paste anything that is not public.

If it does not work

The tool does not show up

In Claude, enable the connector in the conversation from the plus button, then Connectors. In Claude Code, run claude mcp list from the same directory you added it in, then start a new session: a running session does not load a new server.

429 or rate_limited

Each server takes 60 calls a minute from one network, and some tools take fewer. Past that the call comes back as a tool error that starts with rate_limited: wait a minute and ask again. daily_limit means the server reached its shared daily cap, which resets at 00:00 UTC. A limit is not an answer, so do not read it as no results. For higher limits, ask support@openkrill.app for a key.

JSON entry has no type

Claude Code skips an entry in .mcp.json that has a url but no type. Add "type": "http" next to the url, or add the server with claude mcp add --transport http, which writes the type for you. The VS Code entry needs the same type field. Cursor and Pi read the url alone.

A call checked on 2026-10-11

example.com is a public page. The tool reads the head only. The request below was posted to https://sitecheck.openkrill.app/mcp as tools/call. HTTP 200. Source: the Site SEO & AI Crawler Audit server, read 2026-10-11.

{
  "method": "tools/call",
  "params": {
    "name": "check_page_tags",
    "arguments": {
      "url": "https://example.com"
    }
  }
}

• status. 200
• source. The website itself, requested from Agent Tools servers
• notice. This shows what the site publishes to a request from Agent Tools servers at this moment. Firewalls and CDNs can treat real crawlers or visitors differently, and robots.txt is a request that crawlers follow by choice, not
• url. https://example.com/
• finalUrl. https://example.com/

Repeat the call yourself if you need a newer reading. Cached answers expire. A rate limit is not a result: wait and try again. Nothing in the call is a ranking, a filing, or advice.

Using it in production

Nothing to install and nothing to run: no package, no Python or Node.js version and no environment variable. Site SEO & AI Crawler Audit runs on our servers, and your client only needs the URL https://sitecheck.openkrill.app/mcp.

Rate limit	60 tool calls a minute from one network. initialize and tools/list do not count.
Daily cap	5000 tool calls a day per server, shared by every caller. It resets at 00:00 UTC.
At a limit	The call comes back as a tool error that starts with rate_limited (wait a minute) or daily_limit (try after 00:00 UTC), never as a partial answer.
Key	None. Do not add an Authorization header or a key field.
Higher limits	Email support@openkrill.app with the server name, the calls a day you expect and what they are for. We answer within 1 business day.
Freshness	Some answers come from a short cache. Each guide shows the date of its checked call; repeat a call when you need a newer reading.
Uptime	Scheduled test calls check the servers through the same endpoint. There is no SLA yet.
Data	What each server receives and keeps is on the privacy page.

Use it in other apps

• Site SEO & AI Crawler Audit in ChatGPT
• Site SEO & AI Crawler Audit in Cursor
• Site SEO & AI Crawler Audit in VS Code
• Site SEO & AI Crawler Audit in Pi

Roles that use Site SEO & AI Crawler Audit

• SEO specialist guide

Frequently asked questions

How do I install the Site SEO & AI Crawler Audit MCP server in Claude?

In Claude, open Customize, then Connectors, choose Add custom connector and paste https://sitecheck.openkrill.app/mcp. In Claude Code, run claude mcp add --transport http site-check https://sitecheck.openkrill.app/mcp. No restart is needed for either.

Do I need an API key to run Site SEO & AI Crawler Audit in Claude?

No. This server takes no API key. For higher limits, email support@openkrill.app with the server name (Site SEO & AI Crawler Audit), the calls a day you expect and what they are for. We answer within 1 business day.

What should I do if Site SEO & AI Crawler Audit does not connect in Claude?

Check that the connector URL ends in /mcp and that Advanced settings are empty. In Claude Code, run claude mcp get site-check; a JSON entry with a url but no type is skipped, so add type http.