Fairquery Website Tech, SEO & AI-Readiness

Tech stack, SEO score, AI-crawler robots.txt rules, llms.txt, sitemap and security headers; seven free tools.

GitHub

Documentation

Website Tech, SEO & llms.txt Check for Agents

Returns one row per website (tech stack, SEO score, robots.txt AI-crawler rules, sitemap and llms.txt) for $1.50 per 1,000 sites. You pay $0.0015 per site row, and error rows are free. Example: auditing 250 lead websites costs 250 × $0.0015 = $0.38, plus Apify's $0.00005 Actor-start fee.

Fairquery Apify Actor that returns one structured row per website: tech stack, title/meta/Open Graph/JSON-LD, an SEO issue list with a 0–100 score, robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …), XML sitemap, llms.txt / llms-full.txt, security headers, and a 0–100 AI-readiness score. Plain HTTP (no browser), so it's fast and cheap.

When to use this Actor (LLM / tool selection)

Use it when an agent needs to:

  • Enrich a lead/company list with "what is this site built on?" (CMS, framework, analytics, chat, payments, CDN, hosting)
  • Run a quick on-page SEO check of homepages or landing pages
  • Answer "does this site have an llms.txt?" / "does it block GPTBot or ClaudeBot?"
  • Get a site's title, description, social links, feeds and structured-data types before summarizing it

Do not use it for full-site crawls, JS-rendered content (single-page apps may show THIN_CONTENT), Core Web Vitals, or backlink data.

How it works

Per site (max 3 in parallel): GET /robots.txt → GET <url> → sitemap (from robots Sitemap: or /sitemap.xml, /sitemap_index.xml) → /llms.txt and /llms-full.txt. Identifies as FairqueryBot/1.0; by default skips pages that robots.txt disallows for generic crawlers. Tech detection uses ~75 HTML/header/cookie/meta-generator signatures from a single response.

Input

FieldTypeDefaultDescription
urlsstring[]– (required; 3 sample sites are prefilled in the Console form)Domains or URLs
checkRobotsbooltruerobots.txt + AI-bot access map
checkSitemapbooltrueFind sitemap and count URLs
checkLlmsTxtbooltrueCheck /llms.txt and /llms-full.txt
respectRobotsTxtbooltrueSkip URLs disallowed for generic crawlers
maxSitesint500Safety cap per run
{ "urls": ["stripe.com", "https://www.example.org/pricing"], "checkSitemap": false }

Output

Every row has schema_version (1.0.0), type (site | error), source_url (final fetched URL), checked_at.

{
  "type": "site", "domain": "vercel.com", "status_code": 200, "https": true, "response_ms": 310,
  "title": "Agentic Infrastructure - Vercel", "meta_description": "…", "canonical": "https://vercel.com/", "lang": "en",
  "h1": ["…"], "word_count": 256, "open_graph": { "title": "…", "image": "…" }, "json_ld_types": ["Organization", "WebSite"],
  "technologies": [{ "name": "Next.js", "category": "JS framework", "evidence": "html \"/_next/static\"" }, { "name": "Vercel", "category": "Hosting", "evidence": "header x-vercel-id" }],
  "security_headers": { "strict_transport_security": true, "content_security_policy": true },
  "robots_txt_found": true, "ai_bots": { "GPTBot": "partial", "ClaudeBot": "partial" },
  "sitemap_found": true, "sitemap_url_count": 8787, "llms_txt_found": true, "llms_txt_title": "Vercel",
  "seo_score": 100, "seo_issues": [], "ai_readiness_score": 85
}
  • ai_bots[bot]: allowed (no rules), partial (some paths disallowed), blocked (root disallowed).
  • seo_issues[]: {code, severity, message}; codes include NO_HTTPS, MISSING_TITLE, TITLE_TOO_LONG, MISSING_META_DESCRIPTION, MISSING_H1, MULTIPLE_H1, MISSING_CANONICAL, MISSING_LANG, MISSING_VIEWPORT, NOINDEX, MISSING_OPEN_GRAPH, MISSING_STRUCTURED_DATA, IMAGES_MISSING_ALT, THIN_CONTENT, MISSING_ROBOTS_TXT, MISSING_SITEMAP. Score = 100 − 15/8/3 per high/medium/low issue.
  • ai_readiness_score: llms.txt 25, llms-full.txt 5, AI bots allowed up to 20, JSON-LD 15, ≥150 words of server-rendered text 15, sitemap 10, meta description 10; −30 if noindex.

Error rows (machine-readable, never charged)

{ "type": "error", "code": "BLOCKED_OR_RATE_LIMITED", "message": "GET https://x.com/ returned HTTP 403.", "retryable": false, "status_code": 403, "input": "x.com" }

Codes: INPUT_EMPTY, INVALID_URL, MAX_SITES_EXCEEDED, ROBOTS_DISALLOWED, FETCH_FAILED, HTTP_ERROR, BLOCKED_OR_RATE_LIMITED.

Run summary

Key-value store record RUN_SUMMARY: sites checked, errors, HTTP request count, average SEO and AI-readiness scores, llms.txt count, echoed input.

Pricing

EventPriceWhen
site-checked$0.0015Each website row returned
apify-actor-start$0.00005Once per run, per GB of memory (Apify default event)

Error rows (FETCH_FAILED, HTTP_ERROR, BLOCKED_OR_RATE_LIMITED, ROBOTS_DISALLOWED, …) are never charged.

Worked example: 1,000 domains from a lead list where 60 cannot be fetched: 940 × $0.0015 = $1.41, plus $0.00005.

To cap spend, set a maximum cost per run in the run options; the Actor stops cleanly at the limit. maxSites caps how many sites one run checks. Prices are from the Store pricing tab (October 2026).

Memory / runtime

Runs at a fixed 256 MB (default and maximum; the check is network-bound, so more memory does not help), no browser, no proxy. Typically 1–3 s per site, 3 sites in parallel. Slow or rate-limiting sites take longer.

MCP / Apify tools

Call from agents via Apify MCP (https://mcp.apify.com, authenticated with your Apify token), Actor fairquery/website-tech-seo-ai-readiness:

{ "actor": "fairquery/website-tech-seo-ai-readiness", "input": { "urls": ["stripe.com"] } }

Read dataset items (type == "site") and RUN_SUMMARY. A standalone stdio MCP server lives in the GitHub repo under mcp/.

Local stdio MCP server

This repo ships a stdio MCP server (TypeScript, @modelcontextprotocol/sdk) under mcp/ with single-purpose tools. Seven are free and run locally (no token); one optional bulk tool runs this Actor through the Apify API. initialize and tools/list work without a token.

ToolWhat it doesPagingCost
web_quick_check1-20 URLs: status, redirect chain, response time, content type, title–free
web_seo_auditOne URL: meta/OG/JSON-LD/headings/images, seo_issues, seo_score and ai_readiness_score (0-100); output is a reusable snapshot–free
web_ai_crawler_accessrobots.txt rules per AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ...), llms.txt / llms-full.txt–free
web_tech_stackFrameworks, CMS, hosting/CDN, analytics with evidence; category filterlimit/offsetfree
web_sitemap_infoSitemap discovery, index/children, URL list with lastmod; contains, sortlimit/offsetfree
web_security_headersHSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-/Permissions-Policy, COOP; score + grade A-F–free
web_compare_snapshotsDiff two saved snapshots, a saved snapshot vs a fresh re-audit (follow-up), or two live URLs–free
web_batch_auditBulk audit of up to thousands of sites via the Apify Actor (formerly check_websites)–pay-per-event, needs APIFY_TOKEN

Errors are named codes: INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS, SITEMAP_NOT_FOUND, INVALID_ARGUMENTS, APIFY_TOKEN_MISSING, APIFY_ACTOR_RUN_FAILED.

Respectful fetching: User-Agent FairqueryBot ... Fairquery-MCP, 15 s timeout, 1.5 MB HTML / 10 MB sitemap cap, max 5 redirects, and an SSRF guard that refuses hosts (including redirect targets) resolving to private, loopback, link-local or reserved addresses. If you run behind a fake-IP DNS proxy that maps all hosts into 198.18.0.0/15, set FAIRQUERY_ALLOW_FAKE_IP_DNS=1.

Breaking change in v1.2.0: check_websites → web_batch_audit (same input, same billing); check_llms_txt was removed, use web_ai_crawler_access.

cd mcp && npm install && npm run build
npm run test:live                                    # live test of every free tool, no token
node scripts/smoke-test.mjs                          # initialize + tools/list, no token
APIFY_TOKEN=... node scripts/smoke-test.mjs --call   # real web_batch_audit call

{ "mcpServers": { "fairquery-web": { "command": "node", "args": ["/path/to/website-tech-seo-ai-readiness/mcp/dist/index.js"], "env": { "APIFY_TOKEN": "<optional, only for web_batch_audit>" } } } }

Docker: docker build -t fairquery-web-mcp . (the repo-root Dockerfile builds this MCP server and is what Glama uses; the Apify Actor image is built from .actor/Dockerfile). glama.json lists fairquery as maintainer.

Works well after

Troubleshooting

SymptomCause and fix
Run fails at once with INPUT_EMPTYurls was empty. The three sample sites in the Console form are prefill only and are not sent with API calls.
BLOCKED_OR_RATE_LIMITED (HTTP 403/429)The site blocks datacenter traffic or bots. These rows are free; the Actor does not use proxies or browsers to get around blocks.
ROBOTS_DISALLOWEDrobots.txt disallows the page for generic crawlers. Set respectRobotsTxt to false only if you have the right to fetch the page.
Low SEO score or THIN_CONTENT on a modern siteThe page renders its content with JavaScript. The Actor reads the server HTML only (no browser).
Technology missing from the listDetection uses about 75 signatures from a single response, so some tools (for example tag-manager-only scripts) are not visible. The list is heuristic.

FAQ

Does it render JavaScript?

No. It makes plain HTTP requests, which keeps it fast and cheap. Content injected by JavaScript is not seen.

What does the AI-readiness score measure?

llms.txt (25), llms-full.txt (5), AI bots allowed (up to 20), JSON-LD (15), at least 150 words of server-rendered text (15), sitemap (10) and meta description (10); minus 30 for noindex. It is a heuristic, not a ranking factor.

How much does it cost?

$0.0015 per site row ($1.50 per 1,000), plus a $0.00005 start fee. Failed fetches are free.

How do I limit spend?

Set a maximum cost per run in the run options and/or lower maxSites.

Does it crawl the whole site?

No. Per site it fetches robots.txt, the given URL, the sitemap and /llms.txt + /llms-full.txt. Use a crawler for full-site audits.

Can an AI agent call it?

Yes, through Apify MCP with Actor fairquery/website-tech-seo-ai-readiness, or through the stdio MCP server in the GitHub repo, which also has free single-URL tools.

Fairquery family

Other Fairquery tools on the Apify Store. Each one returns structured rows with machine-readable error codes, and error rows are never charged:

All Fairquery Actors: https://apify.com/fairquery

Kits & templates

Optional guides from Fairquery for wiring this Actor into agents and automations (the Actor works fine without them):

Disclaimer

Detection is heuristic (single request, no JS execution). Only publicly served files are read; robots.txt is respected by default.

Links

Development

Local development, tests and deployment notes are in DEVELOPMENT.md. The Apify Store README is .actor/README.md.