SERP Maven

SERP Maven

Self-hosted SERP API alternative: search results from a real Chrome browser and your own residential proxy.

pip install serpmaven. Formerly gserp and google-browser-scraper: both now install serpmaven, and their old commands, imports and GSERP_* / GBS_* variables keep working.

Quickstart · Output · Poisoned results · HTTP API · MCP · Docker · How it works · Options

Scrape Google search results (SERP) with a real Chrome browser and your own residential proxy, and get clean JSON back. Use it as a CLI, a Python library, a local HTTP API, or an MCP server that gives AI agents live Google search.

Plain HTTP is no longer a reliable way to scrape Google, and a fresh browser on a fresh proxy IP is refused quickly. So the scraper warms its first sticky proxy session with ordinary browsing and starts later ones from the cookies Google already accepts, lets the browser pass Google's checks, and then fetches results pages directly over HTTPS with the browser's own cookies. Captchas and blocks are reported as such, never as "no results".

Features

Use cases

• SEO rank tracking and SERP monitoring for your own keywords
• Keyword and competitor research from live Google results
• Web search for AI agents and LLM apps through MCP
• A self-hosted alternative to paid Google search APIs when you already pay for proxies

Install

Python 3.10 or newer. On Windows:

pip install serpmaven

On macOS and Linux the system Python refuses pip install outside a virtual environment, so make one first:

python3 -m venv serpmaven-env
source serpmaven-env/bin/activate
pip install serpmaven

In a new terminal, run source serpmaven-env/bin/activate again before using serpmaven.

The first run offers to download its browser (about 150 MB, once per machine); serpmaven install does it ahead of time, and serpmaven doctor ends with ready when the browser starts and looks like a desktop.

uv tool install serpmaven

sudo apt install xvfb
serpmaven install --with-deps
export SERPMAVEN_BROWSER_ARGS=--enable-unsafe-swiftshader
xvfb-run -a serpmaven doctor

Quickstart

[!NOTE] You need a residential proxy with sticky sessions.

Save your proxy login once. For NodeMaven it asks for the Proxy Username and Proxy Password from the dashboard:

serpmaven login
serpmaven search "best running shoes" --country us -o results.jsonl

Any other provider: save its proxy URL, with {session} where it expects a session id in the username:

serpmaven login --proxy "http://USER-session-{session}:PASS@gate.example.com:7000"
serpmaven search "best running shoes" -o results.jsonl

The first query takes a few minutes while the first proxy IP warms up on a few ordinary pages; the terminal shows each step. Later queries take seconds, and new proxy IPs start from its cookies. Every run ends with the pages served and the traffic used, warm-up apart from the pages.

A file of queries, one per line: serpmaven search -f queries.txt -o results.jsonl. --proxy, --nodemaven, SERPMAVEN_PROXY and NODEMAVEN_LOGIN / NODEMAVEN_PASSWORD win over the saved login; serpmaven logout deletes it. Through NodeMaven, each proxy IP's session id starts with serpmaven, so the traffic shows up as the scraper's on your account; other proxies get the plain id.

[!TIP] 🎁 2 GB of free traffic. No NodeMaven account yet? Sign up and enter the code GOOGLENM. Once per account.

export NODEMAVEN_API_KEY=...
serpmaven search -f queries.txt --nodemaven-demo --traffic-limit-mb 300 -o results.jsonl

serpmaven search "best running shoes" --no-proxy

From Python

from serpmaven import ProxyTemplate, Scraper, Settings

proxy = ProxyTemplate("http://USER-session-{session}:PASS@gate.example.com:7000")
for page in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
    for result in page.get("organic_results", []):
        print(result["position"], result["title"], result["link"])

What you get

SERP Maven returns one JSON object per results page:

{
  "search_metadata": {"status": "Success", "page_verdict": "ok", "fetched_with": "http"},
  "search_parameters": {"engine": "google", "q": "best running shoes", "page": 1},
  "organic_results": [
    {"position": 1, "title": "...", "link": "https://...", "snippet": "..."}
  ],
  "related_questions": [{"question": "..."}],
  "ai_overview": {"text": "..."},
  "exit": {"session": "...", "queries_on_exit": 1, "bytes": "..."}
}

The field names are the ones common to SERP APIs, so simple consumers can often switch with minimal mapping.

Poisoned results protection

Google does not always block a scraper. Sometimes it answers with a normal-looking results page for a different query: no captcha, no error, just wrong data that slips into your pipeline unnoticed.

The scraper checks results pages against the queries it asked. When a proxy IP starts returning unrelated results, it is dropped within a few pages and the query is asked again on a fresh one.

• No extra requests, no extra traffic: the check runs on the page already downloaded
• On by default

Run it as an API

serpmaven serve --country us --prewarm
curl "http://127.0.0.1:8000/search.json?q=best+running+shoes"

In PowerShell, call curl.exe (plain curl there is another command).

• Same record format as above
• --host and --port set the address (default 127.0.0.1:8000)
• pages=N returns N result pages for one query, up to --max-pages (default 3); more is answered 400. One page is a single record, more are a JSON array of records
• hl (language) and gl (country) per request: /search.json?q=...&hl=de&gl=at
• --prewarm finds a proxy IP Google serves at startup, so the first request does not wait for a warm-up; /health answers 503 warming until then
• Results pages are read one at a time on one held proxy IP; a page's links resolve in the background, so the next request starts while the last one waits for its links
• links=N resolves the links of the first N organic results only (0: none)
• links_wait=S answers after S seconds with the links resolved by then; links still being resolved are null and counted in search_metadata.links_pending
• async=true answers at once with search_metadata.id; GET /searches/<id>.json answers "status": "Processing" until the search and its links are done, then the record. The newest 500 searches are kept there, each for up to an hour; up to 50 async searches wait for the browser at once, more are answered 429
• Set SERPMAVEN_API_KEY to require a key; without one the server only listens on localhost

Use it from an AI agent (MCP)

{
  "mcpServers": {
    "google-search": {
      "command": "serpmaven",
      "args": ["mcp", "--country", "us", "--prewarm"]
    }
  }
}

It uses the login saved with serpmaven login; or add "env": {"NODEMAVEN_LOGIN": "...", "NODEMAVEN_PASSWORD": "..."}. One tool, google_search(query, pages, hl, gl, links). It returns the organic results, "People also ask", related searches, and the AI Overview and map places when Google shows them. links is how many of the top results get their full URL (default 5, or --links N on the mcp command); fewer answers sooner, and a result without one carries its site when Google shows it.

Docker

A ready image is published to GitHub Container Registry with every release:

docker run --rm ghcr.io/nodemaven/serpmaven doctor
docker run --rm -p 8000:8000 -e SERPMAVEN_API_KEY=change-me \
  -e SERPMAVEN_PROXY='http://USER-session-{session}:PASS@gate.example.com:7000' \
  ghcr.io/nodemaven/serpmaven serve --host 0.0.0.0

To build it yourself: docker build -t serpmaven .

How it works

1. New identity: a sticky proxy session and a fresh browser profile.
2. Warm-up: the first proxy IP opens a few ordinary pages, staying on each for 20-45 seconds. Later ones start from the Google cookies of a served proxy IP and are warmed up only when no such set is accepted.
3. Search: the first query on a proxy IP goes through the browser. Once Google serves it, the next pages come over HTTPS with the browser's cookies, until Google asks for its check again.
4. Check before parsing: a captcha or a block is reported as one, never parsed as an empty result.
5. Keep or drop: a proxy IP Google answers is reused; one it refuses is dropped and the query retried on a new one. If one proxy IP after another is refused, the run stops instead of spending more traffic.

The defaults come from measurements in proxy-benchmark.

Options

Proxies

Any provider with sticky sessions works through --proxy and a {session} placeholder. A proxy without one is accepted with a warning: if the proxy IP changes on every request, warming it does nothing. SOCKS5 needs an IP-whitelisted endpoint, because Chrome cannot send a SOCKS5 username and password.

Check your machine

serpmaven doctor

Starts the browser on a blank page, sends nothing, and reports what a website would see: whether the browser looks headless, whether WebGL works, the screen size. Google answers some machines much less than others, so run this first.

What this is not

Not a free Google Search API, and not a guarantee that every proxy IP will work. It is a browser-based collector that makes failures explicit, drops proxy IPs Google refuses, and tells you what each page cost.

Status

Early. The cloak engine (--engine cloak) is experimental; patchright is the default.

Google changes its pages often. If results come back empty or wrong, run with --save-html DIR and open an issue with what you see.

You are responsible for using this within Google's terms and the law where you run it.

Adding an engine

An engine joins this tool only after it has been measured on proxy-benchmark. Add it there and run the warm-up ladder against Google:

python scripts/run_ladder.py --engines your-engine

Then open a pull request here with the run file. The ladder runs identities with and without the warm-up in the same hours, so the result shows whether the engine actually gets served, not whether it got lucky once.

License

MIT

_{SERP Maven by NodeMaven}