MyWebSearch

无需 API 密钥！多引擎网页搜索与内容提取 MCP 服务器（Bing、Baidu、DuckDuckGo、GitHub、ArXiv 等），内置反机器人绕过功能。

文档

🔍 MyWebSearch

Keyless Multi-Engine AI Web Search & High-Purity Content Extraction Engine
MCP Server · CLI · Local HTTP Daemon · Skill-Guided Agent Workflows

🇨🇳 简体中文 | 🇺🇸 English

One server, four ways to use it: connect an AI client over MCP, run a one-shot CLI query, keep a local daemon alive, or use the built-in diagnostics before going live.

Jump to: Quick start · MCP tools · CLI · Configuration · Diagnostics · Development

✨ Features

my-websearch is a full-stack search and technical documentation engine built specifically for AI coding agents (Claude Desktop, Cursor, Cherry Studio, Windsurf, Cline, ZCode, etc.). Out of the box, ready for multiple mainstream AI clients:

🌐 16-Engine Federated Search & Smart Routing
Aggregates direct domestic engines (Bing, Bing News, Baidu, CSDN, Juejin, Sogou), global engines (DuckDuckGo, Brave, Startpage, Yahoo, Exa), and specialized developer platforms (GitHub, StackOverflow, Hacker News, arXiv). Features automatic language-aware routing (auto), parallel multi-query execution (queries: string[]), URL deduplication, and automatic minResults cascading fallback. When DuckDuckGo is unhealthy, Yahoo automatically joins the English primary group from the fallback pool.
⚡ Native Web Protocol & Proof-of-Work Solvers (Zero-Browser Fast Path)
Built on wreq-js with native modern TLS/HTTP2 handshake emulation and persistent session cookies. Includes pure-JS/Rust solvers for Startpage's Anubis SHA-256 PoW challenge and DuckDuckGo's d.js (isJsaChallenge) HTML5/arithmetic challenge, resolving structured challenge responses in milliseconds without launching a 400MB browser. Request stacks are engine-specific: DuckDuckGo requires the wreq Chrome profile via GET (axios POST is challenged with HTTP 202), while Brave deliberately uses plain HTTP/1.1 (axios) because CloudFront returns 429 to HTTP/2 impersonation — do not unify them. Note: DuckDuckGo's page-level HTTP 202 anomaly CAPTCHA ("select all squares containing a duck") is human verification and is intentionally not bypassed — it is reported as bot_challenge. Real URL resolution for Sogou redirect links, Baidu Location, and Bing Base64 links.
📄 High-Purity Markdown Article Extraction & Dedicated Site Purifiers
Combines Mozilla Readability with intelligent chrome noise stripping. Features dedicated purification rules for WeChat Official Accounts (mp.weixin.qq.com), StackOverflow questions & accepted answers, and transparent internal routing for CSDN & Juejin. Removes <nav>, <aside>, <footer>, and breadcrumbs while faithfully preserving article titles, code blocks, and GFM tables. Supports seamless pagination (startIndex), plus three token-saving modes: BM25 relevance-ranked focus, CSS-selector extract to JSON with no LLM, and numbered citations.
📚 Unified Context7 Library Docs Lookup (queryDocs)
Directly query official documentation with natural package names (e.g. "next.js", "vue") or Context7 IDs without needing a separate resolve step. Includes automatic candidate ranking, blind-guess 404 self-healing, and related alternatives discovery.
🐙 Extended GitHub / Gitee Resource Fetcher
fetchGithubContent (formerly fetchGithubReadme) supports both GitHub and Gitee: repository READMEs, any source file (/blob/..., with lines/#L120-L180 slicing), directory trees (/tree/...), complete issue / PR discussions, commits (/commit/<sha>), compare views (/compare/a...b), and releases (/releases, /releases/tag/<tag>).
🌏 Zero-Config Network Detection & Fake-IP Ready
Auto-detects OS system proxies (Windows Registry / macOS scutil / Linux env). Automatically routes overseas engines through proxy while keeping domestic engines on fast direct lines. Pre-whitelists 198.18.0.0/15 for TUN / Fake-IP compatibility while enforcing strict SSRF security bounds. No proxy is the supported default: outside mainland China, leave PROXY_URL unset and every engine is queried directly — you don't need Clash/v2ray to use this server.
🔌 Versatile Deployment Modes
Works seamlessly across MCP STDIO (Claude Desktop/Cursor), Streamable HTTP / SSE (Cherry Studio/remote clients), CLI one-shot commands, and a Local HTTP Daemon with connection pool reuse and Prometheus /metrics monitoring.

🌍 16 Search Engines

Engine	Mainland China Connectivity	API Key	Core Technology	Best For
`bing`	🇨🇳 Direct	None	Chrome TLS impersonation + Base64 real URL resolution + stealth fallback	General technical search, mixed EN/ZH queries
`bing-news`	🇨🇳 Direct	None	Bing News RSS + `pubDate` 24h filter + `apiclick.aspx?url=` decoding; opt-in vertical engine	Breaking news, last-24h updates
`baidu`	🇨🇳 Direct	None	Concurrent `Location` redirect resolution + challenge detection	Chinese news, documentation, domestic forums
`csdn`	🇨🇳 Direct	None	WAF cookie jar persistence + empty response auto-retry	Chinese error logs, debugging tutorials
`juejin`	🇨🇳 Direct	None	Direct official API querying for clean long-form articles	Modern frontend/backend technical blogs
`sogou`	🇨🇳 Direct	None	Desktop & mobile dual parsing + redirect link decoding + ad removal	WeChat ecosystem links & long-tail queries
`duckduckgo`	🌐 Proxy in CN	None	wreq Chrome-profile GET of the HTML endpoint (202 anomaly CAPTCHA only on the axios fallback); `d.js` solver retained for the preload fallback	English technical search & open-source discussions
`startpage`	🌐 Proxy in CN	None	Multi-worker Anubis SHA-256 PoW solver (difficulty ≤ 6) + cookie session (no browser needed)	Google-backed high-privacy results
`yahoo`	🌐 Proxy in CN	None	SSR HTML + homepage cookie warm-up + `RU=` redirect decoding; English fallback pool (auto-substitutes when DuckDuckGo is unhealthy)	Stable Bing-index English fallback
`brave`	🌐 Proxy in CN	None	Plain HTTP/1.1 (axios) request path (wreq/HTTP2 impersonation gets 429 from CloudFront) + strict sponsored ad stripping	Independent English index & technical blogs
`exa`	🌐 Proxy preferred	Optional Free Key	Official semantic search API (skipped if key is unset); direct works too, but proxy measured ~0.77s vs ~1.17s direct	Semantic similarity, AI research & papers
`github`	🌐 Direct API	None	Official GitHub Search REST API + exact owner/repo fast path (issues & repositories)	GitHub issues, bug reports, repositories, and exact repo lookup
`stackoverflow`	🌐 Direct API	None	Official StackExchange 2.3 API with HTML entity decoding	Developer troubleshooting, stack traces & answers
`hackernews`	🌐 Direct API	None	Algolia HN Search API with points, comments & author metadata	Tech discussions, startup launches & engineering news
`arxiv`	🌐 Direct API	None	Official arXiv Atom XML API parsing titles, authors & abstracts	Scientific papers, machine learning & AI research
`code`	🌐 Proxy in CN	None	grep.app API search across global GitHub code with repo, path & exact line numbers	Source code implementations, API usage patterns

Version note: this README tracks the current source tree. Yahoo, Bing News, the DDG wreq-GET path and the Brave HTTP/1.1 path ship in 1.3.9; the published npm 1.3.8 still has 14 engines and the older DDG axios POST path.

🛠️ 4 Core MCP Tools (Minimal & Token-Efficient)

💡 Token Budget & Slimming: All tool descriptions are strictly kept within 250 characters (total description characters ≤ 1000). Detailed routing guidance is delegated to McpServer({ instructions }), significantly reducing LLM context consumption.

Tool Name	Purpose	Key Highlights
`search`	Federated web & code search (16 engines)	Supports parallel queries, `timeRange: day\|week\|month\|year` date filter, `site: string[]` filter, `fetchTop` preview injection, `code` engine (grep.app); explicit `engines` defaults to `minResults=0` (no unconstrained cascade); returns inline `diagnostics` on failure
`fetchWebContent`	Web page extraction, Markdown & transparent PDF parsing	All-in-one content extractor: Built-in disk-level Markdown cache (Node 22 `node:sqlite` / partitioned JSON fallback, 200MB LRU, `cache: default\|reload`); transparent PDF parsing (up to 20MB / 50 pages, `[page N]` tags, scanned doc detection); pagination & `focus` keyword targeting (`bm25` relevance-ranked blocks by default, `window` for the legacy fixed window); `selector` to scope extraction to a CSS subtree (errors instead of silently returning the whole page); `extract` for CSS-selector JSON with no LLM; `citations` for numbered references; `retainLinks` to strip URLs (-50% on index pages)
`fetchGithubContent`	Comprehensive GitHub / Gitee content reader	Fetches README, one source file (`/blob/<ref>/<path>`, optional `lines`/`#L120-L180` slicing with a 400-line cap), directory trees (`/tree/`), Issue / PR discussions, commits (`/commit/<sha>`), compare (`/compare/a...b`, capped patch + lockfile filtering), and releases (`/releases[/tag/<tag>]`); `ref` may be a branch, tag, or commit; Gitee parity via API (optional `GITEE_TOKEN`); returns `warnings`
`queryDocs`	All-in-one official library documentation	Resolves library names or Context7 IDs; optional `version` pins Context7/npm/PyPI; 24h local cache (`cache: reload` bypasses it); fallback chain npm → PyPI → crates.io → Go module proxy → GitHub README; returns `source`, `confidence`, and `warnings`

💡 Backward Compatibility: Legacy fetchCsdnArticle and fetchJuejinArticle are automatically routed under fetchWebContent; legacy resolveLibraryId is merged into queryDocs (which accepts a library name or the legacy libraryId alias). The separate legacy tools were removed — the REGISTER_LEGACY_ARTICLE_TOOLS switch is gone, because keeping three unregistered handlers around still cost tool-description budget for nothing.

🚀 Quick Start

1. Run with NPX (Zero Configuration)

No installation required:

npx -y my-websearch@latest

💡 Automated Proxy Discovery: The server automatically detects your operating system's proxy settings. If a local proxy environment is active, overseas engines automatically use the proxy while domestic engines stay on fast direct lines. 198.18.0.0/15 is whitelisted out of the box for Fake-IP environments.

2. Configure in AI Clients

🔹 Automated setup (recommended)

init detects installed MCP clients, writes the server entry into their config files, and keeps a .bak backup:

npx -y my-websearch@latest init                      # auto-detect and write
npx -y my-websearch@latest init --dry-run            # preview the diff, write nothing
npx -y my-websearch@latest init --client cursor,code # target specific clients

Supported out of the box: Claude Desktop, Cursor, and VS Code (workspace .vscode/mcp.json). An existing entry is never overwritten unless you pass --force, corrupt config files are left untouched, and on Windows it writes cmd /c npx for you. Restart the client, then verify with my-websearch doctor.

🔹 Manual configuration

Claude Desktop / Cursor / Windsurf / Cline (STDIO Mode)

Add to your MCP settings file:

{
  "mcpServers": {
    "my-websearch": {
      "command": "npx",
      "args": ["-y", "my-websearch@latest"],
      "env": {
        "MODE": "stdio"
      }
    }
  }
}

(For Windows: init handles this automatically. When configuring manually, if npx is not in the system path, set command to cmd and args to ["/c", "npx", "-y", "my-websearch@latest"])

🔹 Cherry Studio (Streamable HTTP Mode)

Start the server in your terminal with npx my-websearch@latest (default port 3211), then configure:

{
  "mcpServers": {
    "web-search": {
      "name": "MyWebSearch",
      "type": "streamableHttp",
      "baseUrl": "http://localhost:3211/mcp"
    }
  }
}

💻 CLI & Local Daemon

my-websearch can be used directly from your terminal as a standalone CLI tool or persistent daemon. Use --json in automation and keep the daemon on 127.0.0.1 unless you explicitly need remote access:

# Global install
npm install -g my-websearch

# Detect installed MCP clients and write their config (--dry-run to preview)
my-websearch init

# Start persistent local HTTP daemon (port 3210 by default)
my-websearch serve

# One-shot terminal search (reuses running daemon automatically)
my-websearch search "Model Context Protocol best practices" --limit 5 --min-results 5 --json

# Only results published within the last week (day|week|month|year)
my-websearch search "typescript 5.9 release" --time-range week --json

# Extract web page as clean Markdown
my-websearch fetch-web "https://blog.vuejs.org/posts/vue-3-5" --max-chars 15000 --json

# View daemon status and Prometheus metrics (/health, /status, /metrics)
my-websearch status --json

# Clear in-memory search cache
my-websearch cache-clear

🩺 Diagnostics

Use the built-in doctor command before troubleshooting a proxy, browser, or engine:

# Local configuration and browser checks
my-websearch doctor

# Real proxy/engine probes (may contact upstream services)
my-websearch doctor --network

# Limit network probes to selected engines
my-websearch doctor --network --engine bing,duckduckgo --json

# Reset and clear cached engine circuit breaker / health state
my-websearch doctor --reset-health

The report separates configuration, proxy TCP reachability, HTTP reachability, application parsing, browser launch, and challenge verification status. Network mode performs real requests and should not be used as a high-frequency health check.

🌐 Proxy Troubleshooting & Upstream Rate Limiting

When configuring or diagnosing proxy connectivity, refer to these common edge cases:

Startup log shows 🌐 Proxy: disabled even though proxy.url is set?

Check useProxy flag: In ~/.my-websearch/config.json, setting "useProxy": false takes precedence and explicitly overrides and disables url (setting PROXY_URL='false'). Change "useProxy": true or simply omit the field (providing a valid url automatically enables the proxy).
System proxy is auto-detected: when no explicit proxy is configured, the OS system proxy (Windows registry / macOS scutil / HTTPS_PROXY) is used for overseas engines and generic fetches — you do not need PROXY_URL just because a proxy client is running. If detection fails or you want to pin a specific proxy, set PROXY_URL (e.g. http://127.0.0.1:7897).
Prefer ~/.my-websearch/config.json over client env: put proxy.url and API keys in the config file. Some MCP clients (e.g. WorkBuddy) fingerprint the server by command + args + env key names + cwd, so adding a PROXY_URL env entry changes the trust fingerprint and the server may silently drop out of the session. Config-file settings are read at startup and avoid that.

Where to put the proxy (the error message used to say only "MCP client config", which was ambiguous):

Location	Use it when
`~/.my-websearch/config.json` → `"proxy": { "url": "http://127.0.0.1:7897" }`	Recommended. Applies to every client (CLI, daemon, MCP) and does not change client trust fingerprints.
MCP client `env` → `PROXY_URL`	Per-client override. Restart the client; some clients include env keys in their trust fingerprint.
Proxy client's "system proxy" switch	No config at all. Since 1.3.8 it is auto-detected for overseas engines and generic fetches.

Proxy works, but an overseas engine still fails or times out? (Per-engine diagnosis, not a config error)
- DuckDuckGo returns HTTP 202 / challenge: the 202 body is DDG's anomaly CAPTCHA (human verification), not a plain per-IP rate limit; the d.js arithmetic solver does not bypass it. The engine first requests the HTML endpoint with a wreq Chrome-profile GET, which normally returns 200 even when the axios POST path is challenged with 202 — so a 202 now only appears if that path fails and the axios fallback is used. If you still see 202, confirm your build includes the wreq GET path (1.3.8 and earlier do not), then retry or check that the proxy is attached. Note: SEARCH_MODE=playwright:duckduckgo is not supported (DDG has no browser-rendering path), so it falls back to HTTP.
- Bing returns HTTP 200 but 0 results: the wreq impersonation path now falls back to the axios SERP path automatically when it parses an empty page. If both paths stay empty, the query/IP is likely rate-limited.
- Brave returns HTTP 429 Too Many Requests: CloudFront throttles the exit IP for a minutes-long window. It is not permanent — a request after cooldown can return 200 (verified: 429 → later 6/6 × 200 on the same node), but bursts re-trigger it. Measured on the same node: wreq (HTTP/2 Chrome fingerprint) → 429, axios/curl (HTTP/1.1) → 200 — so the current main deliberately uses axios. For a stable path set BRAVE_API_KEY (official API, free tier) and skip the WAF entirely.
- Startpage times out or Anubis PoW difficulty spikes: difficulty 6 is now solved inline by up to 4 worker threads (~3–12s); >6 is skipped. Pass-challenge socket errors / transient 5xx are retried automatically.
- 💡 Resolution: these are upstream anti-bot/rate-limit behaviours, not a broken MCP configuration. Handle them per engine: DDG 202 → upgrade/retry (the wreq GET path; in auto mode Yahoo also replaces an unhealthy DDG from the fallback pool); Brave 429 → wait out the minutes-long window or set BRAVE_API_KEY; Bing 0 results → retry or adjust the query; Startpage PoW → retry. Switch exit IP only after the engine-specific step fails. As a last resort use engines: ["bing", "baidu"], or configure an official key (BRAVE_API_KEY / EXA_API_KEY) for an SLA path.

🔐 Security Model (read before exposing the server)

The defaults are safe as a package, and every one of them matters:

Default	What it protects
Bind to `127.0.0.1`	Nothing outside the machine can reach the port
No authentication	Acceptable only because of loopback binding
Host header validation	Blocks DNS-rebinding attacks from a browser on the same machine
CORS disabled	No cross-origin browser access

Relaxing one without the others silently removes the protection:

HOST=0.0.0.0 (Docker) → the Host allowlist follows the bind address automatically; if you also use a public domain, add it via OPEN_WEBSEARCH_ALLOWED_HOSTS (comma-separated).
Exposing to the public internet → put it behind a reverse proxy with authentication (or an SSH tunnel). The daemon prints a warning when bound to a non-loopback address.
MCP session IDs are identifiers, not credentials — anyone who can reach the port can resume a session.
FETCH_WEB_INSECURE_TLS=true disables certificate verification for user-supplied URLs; only enable it for a trusted network.
The app copies API keys it finds in ~/.gemini/config/mcp_config.json / ~/.workbuddy/mcp.json into ~/.my-websearch/config.json (mode 0600) on first run. Run my-websearch doctor to see where each key came from; set MYWEBSEARCH_NO_AUTO_BOOTSTRAP=1 to disable the copy.

⚙️ Configuration & Environment Variables

Global Configuration (`~/.my-websearch/config.json`)

To share settings across all MCP clients without duplicate configuration:

{
  "apiKeys": {
    "context7": "ctx7sk-...",
    "exa": "..."
  },
  "proxy": {
    "url": "http://127.0.0.1:7897",
    "useProxy": true,
    "engines": ["duckduckgo", "exa", "brave", "startpage"]
  }
}

⚠️ Note on useProxy: When "url" is configured, proxy is enabled automatically. Setting "useProxy": false will explicitly override and disable url. If you want proxy enabled, set "useProxy": true or omit the field entirely.

📋 Exhaustive Environment Variables Reference

Can be set via the global configuration file (~/.my-websearch/config.json), local .env, or system environment variables:

Precedence (highest first): process environment → project .env → project .my-websearch.json → ~/.my-websearch/config.json → built-in defaults.

0. 🖥️ Server & Runtime

Variable	Default	Allowed Values / Description
`MODE`	`both`	`both` / `http` / `stdio`. `http` enables Streamable HTTP + SSE.
`HOST` / `OPEN_WEBSEARCH_HOST`	`127.0.0.1`	MCP HTTP bind address. Binding to a non-loopback address also registers that host in the Host allowlist and prints a no-auth warning.
`PORT`	`3211`	MCP HTTP port
`OPEN_WEBSEARCH_ALLOWED_HOSTS`	empty	Extra Host-header allowlist entries (comma-separated), needed when serving through a public domain/reverse proxy
`OPEN_WEBSEARCH_DAEMON_HOST` / `OPEN_WEBSEARCH_DAEMON_PORT`	`127.0.0.1` / `3210`	Local REST daemon bind address (no auth: keep it on loopback)
`METRICS_ENABLED`	`false`	Enables in-process counters and the daemon `/metrics` endpoint
`FETCH_WEB_DEADLINE_MS`	`45000`	Hard deadline for one `fetchWebContent` call (cookie + browser + retries combined)
`GITHUB_CONTENT_CDN_FIRST`	`false`	Prefer jsDelivr over `raw.githubusercontent.com` for README and source files (public repos only). Legacy alias `GITHUB_README_CDN_FIRST` still works.
`GITHUB_CONTENT_BUDGET_MS`	`12000` (`6000` in CDN-first)	Total time budget for one `fetchGithubContent` call; per-attempt timeout is capped at 4.5s.
`BING_PLAYWRIGHT_BUDGET_MS`	`6000`	Budget for the Bing Playwright fallback before the HTTP results are kept
`MYWEBSEARCH_NO_AUTO_BOOTSTRAP`	unset	Set to `1` to stop copying API keys out of other apps' MCP configs on first run

1. 🔍 Search & Routing Orchestration

Variable	Default	Allowed Values / Description
`DEFAULT_SEARCH_ENGINE`	`auto`	Default engine: `auto` (smart query routing: Chinese `baidu,sogou`, English/foreign `bing,duckduckgo`). Supports all 16 engines: `auto`, `bing`, `bing-news`, `baidu`, `sogou`, `duckduckgo`, `brave`, `csdn`, `juejin`, `startpage`, `yahoo`, `exa`, `github`, `stackoverflow`, `hackernews`, `arxiv`, `code` (vertical developer engines can be configured as default, though `auto` or general web engines are recommended for global default)
`ALLOWED_SEARCH_ENGINES`	all	Comma-separated whitelist of allowed search engines (e.g. `bing,baidu,duckduckgo`)
`BING_BASE_URL`	`https://cn.bing.com/search`	Base URL for Bing queries. Overseas users or reverse proxy nodes can set to `https://www.bing.com/search` (legacy alias `OPEN_WEBSEARCH_BING_HOST` supported)
`DEFAULT_MIN_RESULTS`	`5`	Cascade threshold: triggers fallback engines when usable results are fewer than this value
`DEFAULT_SEARCH_LIMIT`	`10`	Default result limit when not explicitly specified in tool calls (max 50)
`SEARCH_DEADLINE_MS`	`30000`	Global hard deadline for multi-engine search execution (milliseconds)
`SEARCH_FUSION`	`legacy`	Result fusion mode: `legacy` (engine-hit count + arrival order) or `rrf` (Reciprocal Rank Fusion over per-engine ranks, which also replaces the position/consensus terms in re-ranking). RRF stays opt-in until a larger evaluation exists.
`SEARCH_FUSION_RRF_K`	`20`	RRF constant k; lower values amplify rank differences. Only used when `SEARCH_FUSION=rrf`.
`ENGINE_CIRCUIT_BASE_COOLDOWN_MS`	`30000`	First circuit-breaker cooldown for an engine (429 / 202 / bot_challenge). Repeated failures back off exponentially (30s → 60s → 120s…).
`ENGINE_CIRCUIT_MAX_COOLDOWN_MS`	`300000`	Upper bound for the exponential circuit-breaker backoff (default 5 minutes).
`SEARCH_AUTHORITY_DOMAINS`	empty	Additional authority domain whitelist for result re-ranking (comma-separated, appended to built-in list)
`SEARCH_MODE`	`auto`	Search mode and browser policy. Supports single mode, multi-engine per mode, and compound multi-mode combinations: • Global mode: `auto` (default fallback), `request`/`none` (pure HTTP), `playwright` (force browser) • Single mode, multiple engines: `playwright:bing,startpage`, `auto:bing,startpage`, `bing,startpage` • Compound multiple modes: `playwright:bing;auto:startpage` (bing forces browser, startpage uses auto fallback), `auto;request:startpage` (global auto but startpage uses pure HTTP)
`AUTO_ROUTE_EN_ENGINES`	`bing,duckduckgo`	Comma-separated engines for default English auto-routing
`AUTO_ROUTE_ZH_ENGINES`	`baidu,sogou`	Comma-separated engines for default Chinese auto-routing

2. 🌐 Proxy & Network Security

Variable	Default	Allowed Values / Description
`PROXY_URL`	empty (off)	Consolidated upstream proxy configuration (merges previous `USE_PROXY` and `PROXY_ENGINES`). Not needed in TUN mode. • Default proxy: passing empty `PROXY_URL=""` or `PROXY_URL=true` activates default `http://127.0.0.1:7897` (common local proxy ports include 7897, 7890, 10809; overseas engines use proxy, domestic engines stay direct) • Custom address: `PROXY_URL=http://127.0.0.1:10809` • Custom engines: `PROXY_URL="http://127.0.0.1:7897;duckduckgo,brave"` or all engines `PROXY_URL=";all"`
`FAKE_IP_CIDRS`	`198.18.0.0/15`	Whitelisted CIDRs for Fake-IP / TUN mode to avoid SSRF false-positive blocking
`IMPERSONATE_BROWSER`	`chrome_149`	`wreq-js` browser TLS/H2 fingerprint profile (e.g. `chrome_149`, `safari_18`)
`IMPERSONATE_OS`	`windows`	Operating system platform for fingerprint impersonation (`windows`, `macos`, `linux`)
`BING_IMPERSONATE`	`true`	Toggle Chrome TLS impersonation for Bing HTTP mode (auto-trips circuit breaker after consecutive empty results)
`ANUBIS_MAX_POW_DIFFICULTY`	`6`	Maximum difficulty threshold for Startpage Anubis PoW solver; skips/falls back when difficulty exceeds this
`DIRECT_FIRST_PROBE_TIMEOUT_MS`	`3500`	Timeout in ms for direct-first connection probes before proxy fallback
`FETCH_WEB_INSECURE_TLS`	`false`	Disable TLS certificate verification for `fetchWebContent` only (for broken legacy site certs)
`GITHUB_README_CDN_FIRST`	`false`	When `true`, forces skipping `raw.githubusercontent.com` direct probe and queries jsDelivr CDN first (domestic network acceleration)

Custom proxy engine lists: if you set proxy.engines in ~/.my-websearch/config.json or use the ;engine1,engine2 suffix of PROXY_URL, include yahoo. A stale 1.3.7-era list (8 engines, no yahoo) leaves Yahoo on a direct connection — measured from CN: direct times out after 15–20s, while via proxy it returns in ~2s. Removing the list entirely is recommended: defaults now include Yahoo and keep the direct API engines (GitHub / Stack Overflow / Hacker News / arXiv) off the proxy.

3. 🔐 Credentials & API Keys

Variable	Default	Allowed Values / Description
`CONTEXT7_API_KEY`	empty	Context7 API Key (strongly recommended for proxy/TUN users to avoid shared IP quotas)
`EXA_API_KEY`	empty	Exa Free API Key (only required if using the `exa` semantic search engine)
`BRAVE_API_KEY`	empty	Brave Search API Key (optional; uses official high-reliability API when set)
`GITHUB_TOKEN`	empty	GitHub Personal Access Token (optional; raises rate limit to 5,000 req/hr)
`GITEE_TOKEN`	empty	Gitee Personal Access Token (optional; raises Gitee API rate limits and enables private repos)
`STACKOVERFLOW_API_KEY`	empty	Stack Apps Key (optional; raises quota to 10,000 req/day)

4. 🖥️ Server Modes, Concurrency & Session Reaper

Variable	Default	Allowed Values / Description
`MODE`	`both`	MCP transport mode: `stdio` (stdin/stdout only), `http` (HTTP/SSE only), `both` (concurrent)
`PORT`	`3211`	MCP HTTP/SSE listen port (local persistent daemon uses `3210` by default)
`MAX_CONCURRENT_SEARCHES`	`20`	Max concurrent searches in daemon mode (`0` for unlimited)
`MCP_MAX_SESSIONS`	`100`	Max retained MCP HTTP sessions before returning HTTP 429 Too Many Requests
`MCP_SESSION_TTL_MS`	`1800000`	Idle session expiration duration before reaping (default 30 min)
`MCP_SESSION_REAPER_MS`	`300000`	Session reaper scan interval (default 5 min)
`LOG_LEVEL`	`info`	Log level: `quiet` (completely silent), `debug`, `info`, `warn`, `error`
`SECURITY_AUDIT`	`false`	Enable SSRF blocking and host header security audit logging
`OPEN_WEBSEARCH_ALLOWED_HOSTS`	loopback	Host header whitelist for HTTP server (DNS Rebinding protection, comma-separated)

5. 🧭 Playwright Browser & Stealth Controls

[!NOTE] BING_PLAYWRIGHT_FALLBACK and STARTPAGE_PLAYWRIGHT_FALLBACK have been consolidated into SEARCH_MODE (e.g. SEARCH_MODE=bing or SEARCH_MODE=request). Backward compatibility is automatically preserved.

Variable	Default	Allowed Values / Description
`PLAYWRIGHT_CDP_ENDPOINT`	empty	Remote Chrome/Edge CDP endpoint URL (e.g. `http://127.0.0.1:9222` or `ws://...`)
`PLAYWRIGHT_EXECUTABLE_PATH`	auto	Absolute path to browser executable (auto-detects system Edge/Chrome by default)
`PLAYWRIGHT_HEADLESS`	`true`	Run headless (on Windows, automatically utilizes an isolated hidden desktop window)
`PLAYWRIGHT_NAVIGATION_TIMEOUT_MS`	`20000`	Hard page navigation timeout for Playwright (milliseconds)

6. 🛠️ Application Directory & Tool Customization

Variable	Default	Allowed Values / Description
`MYWEBSEARCH_DATA_DIR`	`~/.my-websearch`	Root directory for configuration and persistent cookie files
`OPEN_WEBSEARCH_RESPONSE_CAP_BYTES`	`60000`	Maximum character limit returned to LLMs to prevent prompt context explosion
`MCP_TOOL_SEARCH_NAME`	`search`	Custom registration name for the search tool in MCP
`MCP_TOOL_FETCH_WEB_NAME`	`fetchWebContent`	Custom registration name for the content extraction tool
`MCP_TOOL_FETCH_GITHUB_NAME`	`fetchGithubContent`	Custom registration name for the GitHub/Gitee content tool (set to `fetchGithubReadme` for backward compatibility)

💡 Zero-Config Upgrades (Redundant Variables Removed):

CORS Switches (ENABLE_CORS/CORS_ORIGIN): CORS is disabled by default; enable it explicitly and restrict origins when browser clients need cross-origin access.

Playwright Package Selection (PLAYWRIGHT_PACKAGE/PLAYWRIGHT_MODULE_PATH): Now 100% automated via intelligent client and system binary discovery.

Remote Browser Endpoint (PLAYWRIGHT_WS_ENDPOINT): Unified into standard PLAYWRIGHT_CDP_ENDPOINT.

Emoji Switch (OPEN_WEBSEARCH_NO_EMOJI): Automatic console code page detection smoothly falls back to ASCII tags on legacy terminals.

Fragmented Debug Flag (OPEN_WEBSEARCH_DEBUG): Unified into standard LOG_LEVEL=debug.

Cookie Persistence TTL (MYWEBSEARCH_COOKIE_TTL_DAYS): Solidified to 30 days with automated sliding renewal.

🤝 Contributing & Tests

Pull requests and issues are welcome! To run the test suite locally:

npm install
npm run build
npm test              # Run the Vitest suite (87 files, 667 tests at the current baseline)
npm run test:live     # Run bounded-concurrency live integration tests
npm run eval:live     # Run 23-case live STDIO JSON-RPC evaluation suite

Acknowledgements

Author: wtznicy

This project evolved from Open-WebSearch (originally created by Aas-ee) — special thanks to the original author. Thanks also to:

wreq-js for native Chrome TLS/HTTP2 fingerprint impersonation
Context7 for powering official documentation retrieval
Mozilla Readability & Turndown for content extraction and GFM Markdown formatting