crawlbrulee

the official Model Context Protocol server — @crawlbrulee/mcp. lets mcp-aware agents (Claude Code, Cursor, Codex, Claude Desktop) scrape pages, map sites, and check usage as native tool calls. it wraps @crawlbrulee/sdk, speaks stdio, and is npx -runnable. open source under Apache-2.0 — source on GitHub .

install

Claude Code:

claude mcp add crawlbrulee \
  --env CRAWLBRULEE_API_KEY=cwbl_... \
  -- npx -y @crawlbrulee/mcp

Cursor — add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
    }
  }
}

the same pattern works for Codex, Claude Desktop, and any stdio mcp host: set command: npx, args: ["-y", "@crawlbrulee/mcp"], and forward CRAWLBRULEE_API_KEY in the env block.

as a plugin — the crawlbrulee plugin bundles this server with the agent skills and asks for your key on install. in Claude Code:

/plugin marketplace add crawlbrulee/crawlbrulee-skills
/plugin install crawlbrulee@crawlbrulee

configuration

env var	required	description
CRAWLBRULEE_API_KEY	yes	api key sent as Authorization: Bearer ….

the key is read on the first tool call, so a typo surfaces as a clear tool error rather than a startup failure.

tools

tool	description
scrape	fetch a url → markdown, html, links, images, screenshot, metadata, elements.
scrape_async	submit a scrape job asynchronously; returns a job_id immediately.
scrape_status	check an async job’s status (pending / running / done / failed).
scrape_result	fetch a completed async job’s result (same shape as scrape).
map	enumerate a site’s urls (sitemap + in-page discovery).
usage	current credits, quota, concurrency, reset time. takes no arguments.
whoami	organization + token identity behind the configured key.

example scrape input — only url is required:

{
  "url": "https://example.com",
  "extract": { "markdown": true, "links": true },
  "require_js": false,
  "proxy": "basic"
}

to read named values instead of the whole page, add extract.elements to the scrape or scrape_async input. each key is a name, each value a css selector or an object:

{
  "url": "https://books.toscrape.com/",
  "extract": {
    "elements": {
      "heading": "h1",
      "books": {
        "selector": "article.product_pod",
        "all": true,
        "fields": {
          "title": { "selector": "h3 a", "output": "attribute", "attribute": "title" },
          "price": ".price_color"
        }
      }
    }
  }
}

the values come back in elements, under the same names, at no extra cost. see elements for the request forms, selector rules and limits.

async workflow

use scrape_async instead of scrape for long-running pages (heavy js rendering, full-page screenshots). it takes the same input as scrape, plus an optional webhook: { url, metadata }. poll scrape_status with the returned job_id until the status is done, then call scrape_result to fetch the page. if a webhook was attached on submit, we deliver a single signed scrape.complete POST to that url when the job finishes (with metadata echoed back under data.metadata), so an agent can skip polling entirely.

{ "job_id": "job_abc123" }

usage object

scrape, scrape_result, and map responses (and scrape_status once a job is done) carry a top-level response_meta.usage block, so an agent can report what was delivered and what the call cost. every operation includes { total_credit_cost, engine_credit_cost, proxy_multiplier, engine, proxy }; scrape operations also include screenshot_slicing_credit_cost. map does not, and narrows engine to http | cache.

tool failures return an mcp error result: [<errorName>] <message>, with (HTTP <status>) appended only when an http status applies. if CRAWLBRULEE_API_KEY is unset on the host, calls fail with a synthetic missing_api_key error (rather than a startup failure) — agents can branch on the error code either way.