crawlbrulee

the official Model Context Protocol server, @crawlbrulee/mcp. lets mcp-aware agents (Claude Code, Cursor, Codex, Claude Desktop) scrape pages, map sites, and check usage as native tool calls. it wraps @crawlbrulee/sdk, speaks stdio, and is npx -runnable. open source under Apache-2.0: source on GitHub .

get a key

if you don’t have a crawlbrulee account yet, sign up . the free plan needs no credit card. then, in the dashboard menu, open tokens and click create token.

if the key stops working (the tools answer invalid_credentials), it’s expired, revoked or mistyped. create a new one the same way, replace the old one, and start your agent again.

enter the key yourself, in one of the ways below. don’t paste it into your agent’s chat: anything in the chat stays in its history. your agent can guide you through the setup without seeing the key, then check it with the whoami tool.

install

1. put the key in your environment. add it to your shell profile (for example ~/.zshrc), then open a new terminal and start your agent from it:

export CRAWLBRULEE_API_KEY="cwbl_..."

2. add the server, with no key in its config. it reads the variable each time it starts, so you can also do this step first.

Claude Code:

claude mcp add crawlbrulee -- npx -y @crawlbrulee/mcp

Cursor: add to ~/.cursor/mcp.json (or .cursor/mcp.json in a project). the env line names the variable, so Cursor passes it on:

{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "${env:CRAWLBRULEE_API_KEY}" }
    }
  }
}

Windsurf: add the same entry to its mcp_config.json, which you open from Windsurf’s mcp settings.

Codex: add to ~/.codex/config.toml. Codex passes a variable to the server only when it’s listed in env_vars. codex mcp add crawlbrulee -- npx -y @crawlbrulee/mcp writes the first three lines; add the env_vars line yourself:

[mcp_servers.crawlbrulee]
command = "npx"
args = ["-y", "@crawlbrulee/mcp"]
env_vars = ["CRAWLBRULEE_API_KEY"]

Gemini CLI: add to ~/.gemini/settings.json (or .gemini/settings.json in a project). Gemini CLI passes a variable named like a key only when it’s listed in env, and fills in $CRAWLBRULEE_API_KEY from your shell:

{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "$CRAWLBRULEE_API_KEY" }
    }
  }
}

other stdio mcp hosts: set command: npx and args: ["-y", "@crawlbrulee/mcp"].

apps that don’t read your shell profile

an app opened from the dock or the start menu, like Claude Desktop, may not see the variable, and Claude Desktop’s config can’t name one. there, write the key into the config yourself.

Claude Desktop: open Settings → Developer → Edit Config, or the file itself: ~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows. add:

{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
    }
  }
}

then quit Claude Desktop and open it again.

other ways to enter the key

as a plugin: the crawlbrulee plugin bundles this server with the agent skills and asks for your key on install, in a masked prompt. in Claude Code:

/plugin marketplace add crawlbrulee/crawlbrulee-skills
/plugin install crawlbrulee@crawlbrulee

VS Code can ask for the key in a masked prompt when the server first starts, and store it for you. add to .vscode/mcp.json:

{
  "inputs": [
    {
      "type": "promptString",
      "id": "crawlbrulee-api-key",
      "description": "crawlbrulee api key",
      "password": true
    }
  ],
  "servers": {
    "crawlbrulee": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "${input:crawlbrulee-api-key}" }
    }
  }
}

the key in the config: run this in your own terminal, not through the agent (and not with the ! prefix, which adds the command and its output to the chat):

claude mcp add crawlbrulee \
  --env CRAWLBRULEE_API_KEY=cwbl_... \
  -- npx -y @crawlbrulee/mcp

configuration

env var	required	description
CRAWLBRULEE_API_KEY	yes	api key sent as Authorization: Bearer ….

the key is read on the first tool call, so a typo surfaces as a clear tool error rather than a startup failure.

tools

tool	description
scrape	fetch a url → markdown, html, links, images, screenshot, metadata, elements.
scrape_async	submit a scrape job asynchronously; returns a job_id immediately.
scrape_status	check an async job’s status (pending / running / done / failed).
scrape_result	fetch a completed async job’s result (same shape as scrape).
map	enumerate a site’s urls (sitemap + in-page discovery).
usage	current credits, quota, concurrency, reset time. takes no arguments.
whoami	organization + token identity behind the configured key.

example scrape input. only url is required:

{
  "url": "https://example.com",
  "extract": { "markdown": true, "links": true },
  "require_js": false,
  "proxy": "basic"
}

to read named values instead of the whole page, add extract.elements to the scrape or scrape_async input. each key is a name, each value a css selector or an object:

{
  "url": "https://books.toscrape.com/",
  "extract": {
    "elements": {
      "heading": "h1",
      "books": {
        "selector": "article.product_pod",
        "all": true,
        "fields": {
          "title": { "selector": "h3 a", "output": "attribute", "attribute": "title" },
          "price": ".price_color"
        }
      }
    }
  }
}

the values come back in elements, under the same names, at no extra cost. see elements for the request forms, selector rules and limits.

async workflow

use scrape_async instead of scrape for long-running pages (heavy js rendering, full-page screenshots). it takes the same input as scrape, plus an optional webhook: { url, metadata }. poll scrape_status with the returned job_id until the status is done, then call scrape_result to fetch the page. if a webhook was attached on submit, we deliver a single signed scrape.complete POST to that url when the job finishes (with metadata echoed back under data.metadata), so an agent can skip polling entirely.

{ "job_id": "job_abc123" }

usage object

scrape, scrape_result, and map responses (and scrape_status once a job is done) carry a top-level response_meta.usage block, so an agent can report what was delivered and what the call cost. every operation includes { total_credit_cost, engine_credit_cost, proxy_multiplier, engine, proxy }; scrape operations also include screenshot_slicing_credit_cost. map does not, and narrows engine to http | cache.

tool failures return an mcp error result: [<errorName>] <message>, with (HTTP <status>) appended only when an http status applies. if CRAWLBRULEE_API_KEY is unset on the host, calls fail with a synthetic missing_api_key error (rather than a startup failure), so agents can branch on the error code either way.