MyWebSearch

---

One server, four ways to use it: connect an AI client over MCP, run a one-shot CLI query, keep a local daemon alive, or use the built-in diagnostics before going live.

Jump to: Quick start · MCP tools · CLI · Configuration · Diagnostics · Development

✨ Features

my-websearch is a full-stack search and technical documentation engine built specifically for AI coding agents (Claude Desktop, Cursor, Cherry Studio, Windsurf, Cline, ZCode, etc.). Out of the box, ready for multiple mainstream AI clients:

• 🌐 16-Engine Federated Search & Smart Routing
  Aggregates direct domestic engines (Bing, Baidu, CSDN, Juejin, Sogou), global engines (DuckDuckGo, Brave, Startpage, Yahoo, Exa, Bing News), and specialized developer platforms (GitHub, StackOverflow, Hacker News, arXiv). Features automatic language-aware routing (auto), parallel multi-query execution (queries: string[]), URL deduplication, and automatic minResults cascading fallback. When DuckDuckGo is unhealthy, Yahoo automatically joins the English primary group from the fallback pool.
• ⚡ Native Web Protocol & Proof-of-Work Solvers (Zero-Browser Fast Path)
  Built on wreq-js with native modern TLS/HTTP2 handshake emulation and persistent session cookies. Includes pure-JS/Rust solvers for Startpage's Anubis SHA-256 PoW challenge and DuckDuckGo's d.js (isJsaChallenge) HTML5/arithmetic challenge, resolving structured challenge responses in milliseconds without launching a 400MB browser. Request stacks are engine-specific: DuckDuckGo requires the wreq Chrome profile via GET (axios POST is challenged with HTTP 202), while Brave deliberately uses plain HTTP/1.1 (axios) because CloudFront returns 429 to HTTP/2 impersonation — do not unify them. Note: DuckDuckGo's page-level HTTP 202 anomaly CAPTCHA ("select all squares containing a duck") is human verification and is intentionally not bypassed — it is reported as bot_challenge. Real URL resolution for Sogou redirect links, Baidu Location, and Bing Base64 links.
• 📄 High-Purity Markdown Article Extraction & Dedicated Site Purifiers
  Combines Mozilla Readability with intelligent chrome noise stripping. Features dedicated purification rules for WeChat Official Accounts (mp.weixin.qq.com), StackOverflow questions & accepted answers, and transparent internal routing for CSDN & Juejin. Removes <nav>, <aside>, <footer>, and breadcrumbs while faithfully preserving article titles, code blocks, and GFM tables. Supports seamless pagination (startIndex), plus three token-saving modes: BM25 relevance-ranked focus, CSS-selector extract to JSON with no LLM, and numbered citations.
• 📚 Unified Context7 Library Docs Lookup (queryDocs)
  Directly query official documentation with natural package names (e.g. "next.js", "vue") or Context7 IDs without needing a separate resolve step. Includes automatic candidate ranking, blind-guess 404 self-healing, and related alternatives discovery.
• 🐙 Extended GitHub / Gitee Resource Fetcher
  fetchGithubContent (formerly fetchGithubReadme) supports both GitHub and Gitee: repository READMEs, any source file (/blob/..., with lines/#L120-L180 slicing), directory trees (/tree/...), complete issue / PR discussions, commits (/commit/<sha>), compare views (/compare/a...b), and releases (/releases, /releases/tag/<tag>).
• 🌏 Zero-Config Network Detection & Fake-IP Ready
  Auto-detects OS system proxies (Windows Registry / macOS scutil / Linux env). Automatically routes overseas engines through proxy while keeping domestic engines on fast direct lines. Pre-whitelists 198.18.0.0/15 for TUN / Fake-IP compatibility while enforcing strict SSRF security bounds. No proxy is the supported default: outside mainland China, leave PROXY_URL unset and every engine is queried directly — you don't need Clash/v2ray to use this server.
• 🔌 Versatile Deployment Modes
  Works seamlessly across MCP STDIO (Claude Desktop/Cursor), Streamable HTTP / SSE (Cherry Studio/remote clients), CLI one-shot commands, and a Local HTTP Daemon with connection pool reuse and Prometheus /metrics monitoring.

---

🌍 16 Search Engines

Engine	Mainland China Connectivity	API Key	Core Technology	Best For
bing	🇨🇳 Direct	None	Chrome TLS impersonation + Base64 real URL resolution + stealth fallback	General technical search, mixed EN/ZH queries
bing-news	🌐 Proxy in CN	None	Bing News RSS + pubDate 24h filter + apiclick.aspx?url= decoding; direct can 302 to an HTML page, so it is proxy-eligible by default; opt-in vertical engine	Breaking news, last-24h updates
baidu	🇨🇳 Direct	None	Concurrent Location redirect resolution + challenge detection	Chinese news, documentation, domestic forums
csdn	🇨🇳 Direct	None	WAF cookie jar persistence + empty response auto-retry	Chinese error logs, debugging tutorials
juejin	🇨🇳 Direct	None	Direct official API querying for clean long-form articles	Modern frontend/backend technical blogs
sogou	🇨🇳 Direct	None	Desktop & mobile dual parsing + redirect link decoding + ad removal	WeChat ecosystem links & long-tail queries
duckduckgo	🌐 Proxy in CN	None	wreq Chrome-profile GET of the HTML endpoint (202 anomaly CAPTCHA only on the axios fallback); d.js solver retained for the preload fallback	English technical search & open-source discussions
startpage	🌐 Proxy in CN	None	Multi-worker Anubis SHA-256 PoW solver (difficulty ≤ 6) + cookie session (no browser needed)	Google-backed high-privacy results
yahoo	🌐 Proxy in CN	None	SSR HTML + homepage cookie warm-up + RU= redirect decoding; English fallback pool (auto-substitutes when DuckDuckGo is unhealthy)	Stable Bing-index English fallback
brave	🌐 Proxy in CN	None	Plain HTTP/1.1 (axios) request path (wreq/HTTP2 impersonation gets 429 from CloudFront) + strict sponsored ad stripping	Independent English index & technical blogs
exa	🌐 Proxy preferred	Optional Free Key	Official semantic search API (skipped if key is unset); direct works too, but proxy measured ~0.77s vs ~1.17s direct	Semantic similarity, AI research & papers
github	🌐 Direct API	None	Official GitHub Search REST API + exact owner/repo fast path (issues & repositories)	GitHub issues, bug reports, repositories, and exact repo lookup
stackoverflow	🌐 Direct API	None	Official StackExchange 2.3 API with HTML entity decoding	Developer troubleshooting, stack traces & answers
hackernews	🌐 Direct API	None	Algolia HN Search API with points, comments & author metadata	Tech discussions, startup launches & engineering news
arxiv	🌐 Direct API	None	Official arXiv Atom XML API parsing titles, authors & abstracts	Scientific papers, machine learning & AI research
code	🌐 Proxy in CN	None	grep.app API search across global GitHub code with repo, path & exact line numbers	Source code implementations, API usage patterns

Version note: this README tracks the current source tree (1.3.10). Yahoo, Bing News, the DDG wreq-GET path, the Brave HTTP/1.1 path, and the raw-CDP browser fallback ship in 1.3.10; npm latest may lag until the package is published.

---

🛠️ 4 Core MCP Tools (Minimal & Token-Efficient)

💡 Token Budget & Slimming: All tool descriptions are strictly kept within 250 characters (total description characters ≤ 1000). Detailed routing guidance is delegated to McpServer({ instructions }), significantly reducing LLM context consumption.

Tool Name	Purpose	Key Highlights
search	Federated web & code search (16 engines)	Supports parallel queries, timeRange: day|week|month|year date filter, site: string[] filter, fetchTop preview injection, code engine (grep.app); explicit engines defaults to minResults=0 (no unconstrained cascade); returns inline diagnostics on failure
fetchWebContent	Web page extraction, Markdown & transparent PDF parsing	All-in-one content extractor: Built-in disk-level Markdown cache (Node 22 node:sqlite / partitioned JSON fallback, 200MB LRU, cache: default|reload); transparent PDF parsing (up to 20MB / 50 pages, [page N] tags, scanned doc detection); pagination & focus keyword targeting (bm25 relevance-ranked blocks by default, window for the legacy fixed window); selector to scope extraction to a CSS subtree (errors instead of silently returning the whole page); extract for CSS-selector JSON with no LLM; citations for numbered references; retainLinks to strip URLs (-50% on index pages)
fetchGithubContent	Comprehensive GitHub / Gitee content reader	Fetches README, one source file (/blob/<ref>/<path>, optional lines/#L120-L180 slicing with a 400-line cap), directory trees (/tree/), Issue / PR discussions, commits (/commit/<sha>), compare (/compare/a...b, capped patch + lockfile filtering), and releases (/releases[/tag/<tag>]); ref may be a branch, tag, or commit; Gitee parity via API (optional GITEE_TOKEN); returns warnings
queryDocs	All-in-one official library documentation	Resolves library names or Context7 IDs; optional version pins Context7/npm/PyPI; 24h local cache (cache: reload bypasses it); fallback chain npm → PyPI → crates.io → Go module proxy → GitHub README; returns source, confidence, and warnings

💡 Backward Compatibility: Legacy fetchCsdnArticle and fetchJuejinArticle are automatically routed under fetchWebContent; legacy resolveLibraryId is merged into queryDocs (which accepts a library name or the legacy libraryId alias). The separate legacy tools were removed — the REGISTER_LEGACY_ARTICLE_TOOLS switch is gone, because keeping three unregistered handlers around still cost tool-description budget for nothing.

---

🚀 Quick Start

1. Run with NPX (Zero Configuration)

No installation required:

npx -y my-websearch@latest

💡 Automated Proxy Discovery: The server automatically detects your operating system's proxy settings. If a local proxy environment is active, overseas engines automatically use the proxy while domestic engines stay on fast direct lines. 198.18.0.0/15 is whitelisted out of the box for Fake-IP environments.

2. Configure in AI Clients

🔹 Automated setup (recommended)

init detects installed MCP clients, writes the server entry into their config files, and keeps a .bak backup:

npx -y my-websearch@latest init                      # auto-detect and write
npx -y my-websearch@latest init --dry-run            # preview the diff, write nothing
npx -y my-websearch@latest init --client cursor,code # target specific clients

Supported out of the box: Claude Desktop, Cursor, and VS Code (workspace .vscode/mcp.json). An existing entry is never overwritten unless you pass --force, corrupt config files are left untouched, and on Windows it writes cmd /c npx for you. Restart the client, then verify with my-websearch doctor.

🔹 Manual configuration

Claude Desktop / Cursor / Windsurf / Cline (STDIO Mode)

Add to your MCP settings file:

{
  "mcpServers": {
    "my-websearch": {
      "command": "npx",
      "args": ["-y", "my-websearch@latest"],
      "env": {
        "MODE": "stdio"
      }
    }
  }
}

(For Windows: init handles this automatically. When configuring manually, if npx is not in the system path, set command to cmd and args to ["/c", "npx", "-y", "my-websearch@latest"])

🔹 Cherry Studio (Streamable HTTP Mode)

Start the server in your terminal with npx my-websearch@latest (default port 3211), then configure:

{
  "mcpServers": {
    "web-search": {
      "name": "MyWebSearch",
      "type": "streamableHttp",
      "baseUrl": "http://localhost:3211/mcp"
    }
  }
}

---

💻 CLI & Local Daemon

my-websearch can be used directly from your terminal as a standalone CLI tool or persistent daemon. Use --json in automation and keep the daemon on 127.0.0.1 unless you explicitly need remote access:

# Global install
npm install -g my-websearch

# Detect installed MCP clients and write their config (--dry-run to preview)
my-websearch init

# Start persistent local HTTP daemon (port 3210 by default)
my-websearch serve

# One-shot terminal search (reuses running daemon automatically)
my-websearch search "Model Context Protocol best practices" --limit 5 --min-results 5 --json

# Only results published within the last week (day|week|month|year)
my-websearch search "typescript 5.9 release" --time-range week --json

# Extract web page as clean Markdown
my-websearch fetch-web "https://blog.vuejs.org/posts/vue-3-5" --max-chars 15000 --json

# View daemon status and Prometheus metrics (/health, /status, /metrics)
my-websearch status --json

# Clear in-memory search cache
my-websearch cache-clear

---

🩺 Diagnostics

Use the built-in doctor command before troubleshooting a proxy, browser, or engine:

# Local configuration and browser checks
my-websearch doctor

# Real proxy/engine probes (may contact upstream services)
my-websearch doctor --network

# Limit network probes to selected engines
my-websearch doctor --network --engine bing,duckduckgo --json

# Reset and clear cached engine circuit breaker / health state
my-websearch doctor --reset-health

The report separates configuration, proxy TCP reachability, HTTP reachability, application parsing, browser launch, and challenge verification status. Network mode performs real requests and should not be used as a high-frequency health check.

🌐 Proxy Troubleshooting & Upstream Rate Limiting

When configuring or diagnosing proxy connectivity, refer to these common edge cases:

1. Startup log shows 🌐 Proxy: disabled even though proxy.url is set?

   • Check useProxy flag: In ~/.my-websearch/config.json, setting "useProxy": false takes precedence and explicitly overrides and disables url (setting PROXY_URL='false'). Change "useProxy": true or simply omit the field (providing a valid url automatically enables the proxy).
   • System proxy is auto-detected: when no explicit proxy is configured, the OS system proxy (Windows registry / macOS scutil / HTTPS_PROXY) is used for overseas engines and generic fetches — you do not need PROXY_URL just because a proxy client is running. If detection fails or you want to pin a specific proxy, set PROXY_URL (e.g. http://127.0.0.1:7897).
   • Prefer ~/.my-websearch/config.json over client env: put proxy.url and API keys in the config file. Some MCP clients (e.g. WorkBuddy) fingerprint the server by command + args + env key names + cwd, so adding a PROXY_URL env entry changes the trust fingerprint and the server may silently drop out of the session. Config-file settings are read at startup and avoid that.
   • Where to put the proxy (the error message used to say only "MCP client config", which was ambiguous):

     Location	Use it when
     ~/.my-websearch/config.json → "proxy": { "url": "http://127.0.0.1:7897" }	Recommended. Applies to every client (CLI, daemon, MCP) and does not change client trust fingerprints.
     MCP client env → PROXY_URL	Per-client override. Restart the client; some clients include env keys in their trust fingerprint.
     Proxy client's "system proxy" switch	No config at all. Since 1.3.8 it is auto-detected for overseas engines and generic fetches.

2. Proxy works, but an overseas engine still fails or times out? (Per-engine diagnosis, not a config error)

   • DuckDuckGo returns HTTP 202 / challenge: the 202 body is DDG's anomaly CAPTCHA (human verification), not a plain per-IP rate limit; the d.js arithmetic solver does not bypass it. The engine first requests the HTML endpoint with a wreq Chrome-profile GET, which normally returns 200 even when the axios POST path is challenged with 202 — so a 202 now only appears if that path fails and the axios fallback is used. If you still see 202, confirm your build includes the wreq GET path (1.3.8 and earlier do not), then retry or check that the proxy is attached. DDG remains best-effort: the same exit can still be challenged intermittently, so do not treat it as an SLA; for stability use Bing/Startpage or configure BRAVE_API_KEY/EXA_API_KEY. Note: SEARCH_MODE=playwright:duckduckgo is not supported (DDG has no browser-rendering path), so it falls back to HTTP.
   • Bing returns HTTP 200 but 0 results: the wreq impersonation path now falls back to the axios SERP path automatically when it parses an empty page. If both paths stay empty, the query/IP is likely rate-limited.
   • Brave returns HTTP 429 Too Many Requests: CloudFront throttles the exit IP for a minutes-long window. It is not permanent — a request after cooldown can return 200 (verified: 429 → later 6/6 × 200 on the same node), but bursts re-trigger it. Measured on the same node: wreq (HTTP/2 Chrome fingerprint) → 429, axios/curl (HTTP/1.1) → 200 — so the current main deliberately uses axios. For a stable path set BRAVE_API_KEY (official API, free tier) and skip the WAF entirely.
   • Startpage times out or Anubis PoW difficulty spikes: difficulty 6 is now solved inline by up to 4 worker threads (~3–12s); >6 is skipped. Pass-challenge socket errors / transient 5xx are retried automatically.
   • 💡 Resolution: these are upstream anti-bot/rate-limit behaviours, not a broken MCP configuration. Handle them per engine: DDG 202 → upgrade/retry (the wreq GET path; in auto mode Yahoo also replaces an unhealthy DDG from the fallback pool); Brave 429 → wait out the minutes-long window or set BRAVE_API_KEY; Bing 0 results → retry or adjust the query; Startpage PoW → retry. Switch exit IP only after the engine-specific step fails. As a last resort use engines: ["bing", "baidu"], or configure an official key (BRAVE_API_KEY / EXA_API_KEY) for an SLA path.

🔐 Security Model (read before exposing the server)

The defaults are safe as a package, and every one of them matters:

Default	What it protects
Bind to 127.0.0.1	Nothing outside the machine can reach the port
No authentication	Acceptable only because of loopback binding
Host header validation	Blocks DNS-rebinding attacks from a browser on the same machine
CORS disabled	No cross-origin browser access

Relaxing one without the others silently removes the protection:

• HOST=0.0.0.0 (Docker) → the Host allowlist follows the bind address automatically; if you also use a public domain, add it via OPEN_WEBSEARCH_ALLOWED_HOSTS (comma-separated).
• Exposing to the public internet → put it behind a reverse proxy with authentication (or an SSH tunnel). The daemon prints a warning when bound to a non-loopback address.
• MCP session IDs are identifiers, not credentials — anyone who can reach the port can resume a session.
• FETCH_WEB_INSECURE_TLS=true disables certificate verification for user-supplied URLs; only enable it for a trusted network.
• The app copies API keys it finds in ~/.gemini/config/mcp_config.json / ~/.workbuddy/mcp.json into ~/.my-websearch/config.json (mode 0600) on first run. Run my-websearch doctor to see where each key came from; set MYWEBSEARCH_NO_AUTO_BOOTSTRAP=1 to disable the copy.

⚙️ Configuration & Environment Variables

Global Configuration (~/.my-websearch/config.json)

To share settings across all MCP clients without duplicate configuration:

{
  "apiKeys": {
    "context7": "ctx7sk-...",
    "exa": "..."
  },
  "proxy": {
    "url": "http://127.0.0.1:7897",
    "useProxy": true,
    "engines": ["duckduckgo", "exa", "brave", "startpage"]
  }
}

⚠️ Note on useProxy: When "url" is configured, proxy is enabled automatically. Setting "useProxy": false will explicitly override and disable url. If you want proxy enabled, set "useProxy": true or omit the field entirely.

📋 Exhaustive Environment Variables Reference

Can be set via the global configuration file (~/.my-websearch/config.json), local .env, or system environment variables:

Precedence (highest first): process environment → project .env → project .my-websearch.json → ~/.my-websearch/config.json → built-in defaults.

0. 🖥️ Server & Runtime

Variable	Default	Allowed Values / Description
MODE	both	both / http / stdio. http enables Streamable HTTP + SSE.
HOST / OPEN_WEBSEARCH_HOST	127.0.0.1	MCP HTTP bind address. Binding to a non-loopback address also registers that host in the Host allowlist and prints a no-auth warning.
PORT	3211	MCP HTTP port
OPEN_WEBSEARCH_ALLOWED_HOSTS	empty	Extra Host-header allowlist entries (comma-separated), needed when serving through a public domain/reverse proxy
OPEN_WEBSEARCH_DAEMON_HOST / OPEN_WEBSEARCH_DAEMON_PORT	127.0.0.1 / 3210	Local REST daemon bind address (no auth: keep it on loopback)
METRICS_ENABLED	false	Enables in-process counters and the daemon /metrics endpoint
FETCH_WEB_DEADLINE_MS	45000	Hard deadline for one fetchWebContent call (cookie + browser + retries combined)
GITHUB_CONTENT_CDN_FIRST	false	Prefer jsDelivr over raw.githubusercontent.com for README and source files (public repos only). Legacy alias GITHUB_README_CDN_FIRST still works.
GITHUB_CONTENT_BUDGET_MS	12000 (6000 in CDN-first)	Total time budget for one fetchGithubContent call; per-attempt timeout is capped at 4.5s.
BING_PLAYWRIGHT_BUDGET_MS	6000	Legacy name retained for compatibility: budget for the Bing browser fallback (now backed by raw CDP) before HTTP results are kept
MYWEBSEARCH_NO_AUTO_BOOTSTRAP	unset	Set to 1 to stop copying API keys out of other apps' MCP configs on first run

1. 🔍 Search & Routing Orchestration

Variable	Default	Allowed Values / Description
DEFAULT_SEARCH_ENGINE	auto	Default engine: auto (smart query routing: Chinese baidu,sogou, English/foreign bing,duckduckgo). Supports all 16 engines: auto, bing, bing-news, baidu, sogou, duckduckgo, brave, csdn, juejin, startpage, yahoo, exa, github, stackoverflow, hackernews, arxiv, code (vertical developer engines can be configured as default, though auto or general web engines are recommended for global default)
ALLOWED_SEARCH_ENGINES	all	Comma-separated whitelist of allowed search engines (e.g. bing,baidu,duckduckgo)
BING_BASE_URL	https://cn.bing.com/search	Base URL for Bing queries. Overseas users or reverse proxy nodes can set to https://www.bing.com/search (legacy alias OPEN_WEBSEARCH_BING_HOST supported)
DEFAULT_MIN_RESULTS	5	Cascade threshold: triggers fallback engines when usable results are fewer than this value
DEFAULT_SEARCH_LIMIT	10	Default result limit when not explicitly specified in tool calls (max 50)
SEARCH_DEADLINE_MS	30000	Global hard deadline for multi-engine search execution (milliseconds)
SEARCH_FUSION	legacy	Result fusion mode: legacy (engine-hit count + arrival order) or rrf (Reciprocal Rank Fusion over per-engine ranks, which also replaces the position/consensus terms in re-ranking). RRF stays opt-in until a larger evaluation exists.
SEARCH_FUSION_RRF_K	20	RRF constant k; lower values amplify rank differences. Only used when SEARCH_FUSION=rrf.
ENGINE_CIRCUIT_BASE_COOLDOWN_MS	30000	First circuit-breaker cooldown for an engine (429 / 202 / bot_challenge). Repeated failures back off exponentially (30s → 60s → 120s…).
ENGINE_CIRCUIT_MAX_COOLDOWN_MS	300000	Upper bound for the exponential circuit-breaker backoff (default 5 minutes).
SEARCH_AUTHORITY_DOMAINS	empty	Additional authority domain whitelist for result re-ranking (comma-separated, appended to built-in list)
SEARCH_MODE	auto	Search mode and browser policy. Supports single mode, multi-engine per mode, and compound multi-mode combinations: • Global mode: auto (default fallback), request/none (pure HTTP), playwright (force browser; legacy value name, now backed by CDP) • Single mode, multiple engines: playwright:bing,startpage, auto:bing,startpage, bing,startpage • Compound multiple modes: playwright:bing;auto:startpage (bing forces browser, startpage uses auto fallback), auto;request:startpage (global auto but startpage uses pure HTTP)
AUTO_ROUTE_EN_ENGINES	bing,duckduckgo	Comma-separated engines for default English auto-routing
AUTO_ROUTE_ZH_ENGINES	baidu,sogou	Comma-separated engines for default Chinese auto-routing

2. 🌐 Proxy & Network Security

Variable	Default	Allowed Values / Description
PROXY_URL	empty (off)	Consolidated upstream proxy configuration (merges previous USE_PROXY and PROXY_ENGINES). Not needed in TUN mode. • Default proxy: passing empty PROXY_URL="" or PROXY_URL=true activates default http://127.0.0.1:7897 (common local proxy ports include 7897, 7890, 10809; overseas engines use proxy, domestic engines stay direct) • Custom address: PROXY_URL=http://127.0.0.1:10809 • Custom engines: PROXY_URL="http://127.0.0.1:7897;duckduckgo,brave" or all engines PROXY_URL=";all"
FAKE_IP_CIDRS	198.18.0.0/15	Whitelisted CIDRs for Fake-IP / TUN mode to avoid SSRF false-positive blocking
IMPERSONATE_BROWSER	chrome_149	wreq-js browser TLS/H2 fingerprint profile (e.g. chrome_149, safari_18)
IMPERSONATE_OS	windows	Operating system platform for fingerprint impersonation (windows, macos, linux)
BING_IMPERSONATE	true	Toggle Chrome TLS impersonation for Bing HTTP mode (auto-trips circuit breaker after consecutive empty results)
ANUBIS_MAX_POW_DIFFICULTY	6	Maximum difficulty threshold for Startpage Anubis PoW solver; skips/falls back when difficulty exceeds this
DIRECT_FIRST_PROBE_TIMEOUT_MS	3500	Timeout in ms for direct-first connection probes before proxy fallback
FETCH_WEB_INSECURE_TLS	false	Disable TLS certificate verification for fetchWebContent only (for broken legacy site certs)
GITHUB_README_CDN_FIRST	false	When true, forces skipping raw.githubusercontent.com direct probe and queries jsDelivr CDN first (domestic network acceleration)

Custom proxy engine lists: if you set proxy.engines in ~/.my-websearch/config.json or use the ;engine1,engine2 suffix of PROXY_URL, include yahoo. A stale 1.3.7-era list (8 engines, no yahoo) leaves Yahoo on a direct connection — measured from CN: direct times out after 15–20s, while via proxy it returns in ~2s. Removing the list entirely is recommended: defaults now include Yahoo and keep the direct API engines (GitHub / Stack Overflow / Hacker News / arXiv) off the proxy.

3. 🔐 Credentials & API Keys

Variable	Default	Allowed Values / Description
CONTEXT7_API_KEY	empty	Context7 API Key (strongly recommended for proxy/TUN users to avoid shared IP quotas)
EXA_API_KEY	empty	Exa Free API Key (only required if using the exa semantic search engine)
BRAVE_API_KEY	empty	Brave Search API Key (optional; uses official high-reliability API when set)
GITHUB_TOKEN	empty	GitHub Personal Access Token (optional; raises rate limit to 5,000 req/hr)
GITEE_TOKEN	empty	Gitee Personal Access Token (optional; raises Gitee API rate limits and enables private repos)
STACKOVERFLOW_API_KEY	empty	Stack Apps Key (optional; raises quota to 10,000 req/day)

4. 🖥️ Server Modes, Concurrency & Session Reaper

Variable	Default	Allowed Values / Description
MODE	both	MCP transport mode: stdio (stdin/stdout only), http (HTTP/SSE only), both (concurrent)
PORT	3211	MCP HTTP/SSE listen port (local persistent daemon uses 3210 by default)
MAX_CONCURRENT_SEARCHES	20	Max concurrent searches in daemon mode (0 for unlimited)
MCP_MAX_SESSIONS	100	Max retained MCP HTTP sessions before returning HTTP 429 Too Many Requests
MCP_SESSION_TTL_MS	1800000	Idle session expiration duration before reaping (default 30 min)
MCP_SESSION_REAPER_MS	300000	Session reaper scan interval (default 5 min)
LOG_LEVEL	info	Log level: quiet (completely silent), debug, info, warn, error
SECURITY_AUDIT	false	Enable SSRF blocking and host header security audit logging
OPEN_WEBSEARCH_ALLOWED_HOSTS	loopback	Host header whitelist for HTTP server (DNS Rebinding protection, comma-separated)

5. 🧭 CDP Browser & Stealth Controls (legacy Playwright env names kept)

[!NOTE] The browser fallback now uses raw Chrome DevTools Protocol via chrome-remote-interface; it launches the Chrome/Edge already installed on your system. The PLAYWRIGHT_* env names and SEARCH_MODE=playwright value are kept for backward compatibility only. BING_PLAYWRIGHT_FALLBACK and STARTPAGE_PLAYWRIGHT_FALLBACK have been consolidated into SEARCH_MODE.

Variable	Default	Allowed Values / Description
BROWSER_DRIVER	cdp	Browser backend. cdp (default, raw CDP). playwright is a legacy escape hatch and requires you to install playwright-core manually; it is no longer bundled.
PLAYWRIGHT_CDP_ENDPOINT	empty	Remote Chrome/Edge CDP endpoint URL (e.g. http://127.0.0.1:9222 or ws://...); legacy name, used by the CDP driver
PLAYWRIGHT_EXECUTABLE_PATH	auto	Absolute path to the browser executable (auto-detects system Edge/Chrome by default); legacy name
PLAYWRIGHT_HEADLESS	true	Run headless (on Windows, an isolated hidden desktop window is used); legacy name
PLAYWRIGHT_NAVIGATION_TIMEOUT_MS	20000	Hard page navigation timeout (milliseconds); legacy name, applied to CDP navigation

6. 🛠️ Application Directory & Tool Customization

Variable	Default	Allowed Values / Description
MYWEBSEARCH_DATA_DIR	~/.my-websearch	Root directory for configuration and persistent cookie files
OPEN_WEBSEARCH_RESPONSE_CAP_BYTES	60000	Maximum character limit returned to LLMs to prevent prompt context explosion
MCP_TOOL_SEARCH_NAME	search	Custom registration name for the search tool in MCP
MCP_TOOL_FETCH_WEB_NAME	fetchWebContent	Custom registration name for the content extraction tool
MCP_TOOL_FETCH_GITHUB_NAME	fetchGithubContent	Custom registration name for the GitHub/Gitee content tool (set to fetchGithubReadme for backward compatibility)

💡 Zero-Config Upgrades (Redundant Variables Removed):

• CORS Switches (ENABLE_CORS/CORS_ORIGIN): CORS is disabled by default; enable it explicitly and restrict origins when browser clients need cross-origin access.
• Browser Backend: raw CDP via chrome-remote-interface; playwright-core is no longer bundled. PLAYWRIGHT_* names and SEARCH_MODE=playwright remain as compatibility aliases.
• Emoji Switch (OPEN_WEBSEARCH_NO_EMOJI): Automatic console code page detection smoothly falls back to ASCII tags on legacy terminals.
• Fragmented Debug Flag (OPEN_WEBSEARCH_DEBUG): Unified into standard LOG_LEVEL=debug.
• Cookie Persistence TTL (MYWEBSEARCH_COOKIE_TTL_DAYS): Solidified to 30 days with automated sliding renewal.

---

🤝 Contributing & Tests

Pull requests and issues are welcome! To run the test suite locally:

npm install
npm run build
npm test              # Run the Vitest suite (88 files, 684 tests at the current baseline)
npm run test:live     # Run bounded-concurrency live integration tests
npm run eval:live     # Run 23-case live STDIO JSON-RPC evaluation suite

Acknowledgements

Author: wtznicy

This project evolved from Open-WebSearch (originally created by Aas-ee) — special thanks to the original author. Thanks also to:

• wreq-js for native Chrome TLS/HTTP2 fingerprint impersonation
• Context7 for powering official documentation retrieval
• Mozilla Readability & Turndown for content extraction and GFM Markdown formatting