web-search-mcp

OmniSearch

A comprehensive Model Context Protocol (MCP) server built with FastMCP that provides LLMs with real-time, high-fidelity access to the web. This server aggregates multiple search engines, social platforms, and developer tools into a single interface, allowing AI agents to perform deep research, track community sentiment, and analyze technical documentation.

Design docs → wiki — tool selection guide, decision matrix, recommended workflows, tools status & known quirks, plugin setup, and development standards.

---

🚀 Features

The server provides a diverse suite of tools categorized by their primary use case:

🌐 General Web Search & Retrieval

Tool	Description	Best For
search_web	Fast web search via DuckDuckGo or Exa (SDK). Supports domain-scoping, date filtering, news mode, and geographic region. Default auto-provider tries DDG first, falls back to Exa.	Quick lookups, high-volume searches, pagination, broad coverage
fetch_page	High-fidelity text extraction from URLs with bot-detection bypass, SSRF protection (blocks private/internal IPs), and multiple output formats.	Deep reading of search results, standalone URL fetching

💬 Social & Community Intelligence

Tool	Description	Best For
search_reddit	Keyless search for community discussions, opinions, and real-world user experiences via RSS + Shreddit enrichment + Arctic-Shift score backfill.	Product reviews, community sentiment, troubleshooting
search_hackernews	Technical discourse, startup news, and developer opinions via the Algolia HN API.	Tech news, startup discussions, developer opinions
search_github	Search for Issues and PRs to track bugs, feature requests, and community sentiment. Requires gh CLI or GITHUB_TOKEN.	Bug tracking, feature requests, community sentiment
get_github_issue	Fetch full conversation threads from GitHub Issues/PRs, sorted by reactions with author/date/reactions metadata.	Deep-diving into specific issues/PRs
search_x	Real-time discourse and breaking news via Xquik API or vendored Bird CLI (requires session cookies or API key).	Breaking news, community reactions, engagement signals
search_linkedin	Search people, companies, jobs, posts via DuckDuckGo + Jina Reader (r.jina.ai). No API key needed.	Professional profiles, company research, job search
search_digg	Curated AI discourse clusters from ~1000 high-signal X accounts via digg-pp-cli. TLDRs + top X posts per cluster.	AI trends, curated high-signal discourse
search_techmeme	Editorial tech news from live archive (2005+) via techmeme-pp-cli. Headlines with source attribution.	Tech industry news, major announcements
search_polymarket	Prediction markets with real-money odds via public Gamma API. Multi-query expansion + acronym credit.	Forecasting, sentiment, real-money odds on events

🎓 Academic & Reference

Tool	Description	Best For
search_arxiv	Specialized search for academic papers with Lucene field prefixes (au:, ti:, cat:, abs:).	Research papers, citations, literature reviews
search_wikipedia	Factual summaries and background research via the MediaWiki API.	Factual summaries, background research, citations
rerank_results	Rerank candidate texts against a query by relevance. Three backends: token_overlap (default, zero-dep), flashrank (neural ONNX cross-encoder), hybrid (blend). Supports metadata boosts (engagement/freshness/authority), deduplication, and chunk diversity.	Post-search relevance refinement, LLM context prioritization

---

📋 Prerequisites

Requirement	Version	Notes
Python	3.11+	Required
uv	Latest	Recommended for installation and environment management

Optional External Tools

Tool	Required For	Installation
gh CLI	Authenticated GitHub search & issue retrieval (higher rate limits)	brew install gh / github.com/cli/cli
Node.js	Vendored Bird CLI for X/Twitter search (not needed with XQUIK_API_KEY)	22+ recommended; brew install node@22 / nodejs.org
digg-pp-cli	Digg AI 1000 curated clusters (free, no auth)	npx -y @mvanhorn/printing-press-library install digg --cli-only
techmeme-pp-cli	Techmeme tech news archive search (free, no auth)	npx -y @mvanhorn/printing-press-library install techmeme --cli-only
pdftotext	Local corpus PDF extraction (optional, for future --corpus flag)	brew install poppler / apt install poppler-utils

Optional Python Extras

Extra	Install Command	Enables
flashrank	uv pip install 'web_search_mcp[flashrank]'	Neural reranking backend (~23MB ONNX, CPU, no torch)
transformers	uv pip install 'web_search_mcp[transformers]'	sentence-transformers backend (requires torch)
eval	uv pip install 'web_search_mcp[eval]'	BM25/sklearn evaluation utilities

---

⚙️ Installation

You have three options depending on your use case:

Option A: Quick Run (via uvx)

Fastest way to try it out without cloning the repo. Add to your MCP client config:

{
  "mcpServers": {
    "OmniSearch": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/sydasif/web-search-mcp.git",
        "omnisearch"
      ]
    }
  }
}

Option B: Permanent Install

Fastest startup times with a globally installed tool:

uv tool install git+https://github.com/sydasif/web-search-mcp.git

Then configure your MCP client:

{
  "mcpServers": {
    "OmniSearch": {
      "command": "omnisearch"
    }
  }
}

Option C: Development Install

If you want to modify the code or contribute:

git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync
uv run omnisearch

Verify It's Working

Once the server is running, try a simple search:

search_web(query="current weather in Tokyo")

---

Option D: Plugin Install

Install as a plugin for quick access from any project:

claude plugin marketplace add sydasif/web-search-mcp
claude plugin install --scope user web-search@sydasif-web-search-mcp

OMP Plugin Install

omp plugin marketplace add sydasif/web-search-mcp
omp plugin install --scope user web-search@sydasif-web-search-mcp

🔐 Configuration & Authentication

Most tools work out of the box with zero configuration. The following environment variables are only needed for premium or authenticated features.

Environment Variables Reference

Variable	Required For	How to Get It
EXA_API_KEY	Exa AI semantic search (optional fallback)	Sign up at exa.ai
GITHUB_TOKEN	Higher GitHub API rate limits (optional)	Generate a GitHub PAT
AUTH_TOKEN	X/Twitter search via Bird CLI (required)	Session cookie from x.com (see below)
CT0	X/Twitter search via Bird CLI (required)	Session cookie from x.com (see below)
XQUIK_API_KEY	X/Twitter search via Xquik API (alternative to cookies)	Sign up at xquik.ai
SEARCH_MCP_SEARXNG_URL	Optional SearXNG instance for keyless web search fallback	Your SearXNG instance URL
SEARCH_MCP_CORPUS_DIRS	Local document directories for private search (colon-separated paths) — planned	Paths to your .md/.txt/.pdf directories
SEARCH_MCP_DEFAULT_SOURCES	Fixed default source set (comma-separated, e.g. reddit,x,github)	Source names from available sources
SEARCH_MCP_INCLUDE_SOURCES	Opt-in additive sources (e.g. perplexity,linkedin)	Source names
SEARCH_MCP_EXCLUDE_SOURCES	Hard-exclude sources (e.g. x,reddit)	Source names
SEARCH_MCP_VERIFY_FRESHNESS	Enable default fact-checking for Polymarket odds, GitHub stars, etc. — planned	true or false

Setting Up GitHub Authentication

Option 1 — Recommended: Use gh CLI

gh auth login

The server detects your local session automatically.

Option 2: Manual Token

export GITHUB_TOKEN="ghp_your_token_here"

Setting Up X/Twitter Authentication

X/Twitter search requires either session cookies or an API key.

Option 1 — Session Cookies (Bird CLI):

1. Log into x.com in your browser.
2. Open DevTools (F12) → Application (or Storage) → Cookies → x.com.
3. Copy the values for auth_token and ct0.
4. Export them in the shell where the MCP server runs:

   export AUTH_TOKEN="your_auth_token"
   export CT0="your_ct0"
   

   Note: These are session cookies. If searches return 401s, refresh them by logging out and back in.

Option 2 — Xquik API Key (Recommended):

1. Sign up at xquik.ai to get an API key.
2. Export it:

   export XQUIK_API_KEY="your_xquik_key"
   

   This bypasses the Node.js Bird CLI dependency entirely.

Setting Up Exa AI (Optional)

Exa provides semantic search and JS-heavy page fallback:

export EXA_API_KEY="your_exa_key"

---

💡 Usage Examples

Web Research

# Broad search (auto: DDG first, falls back to Exa on error or zero results)
search_web(query="Latest NVIDIA H200 benchmarks")

# Force DDG explicitly
search_web(query="uv package manager", provider="ddg")

# Force Exa explicitly
search_web(query="uv package manager", provider="exa")

# Targeted documentation search
search_web(query="useEffect cleanup", domain="react.dev")

# News with region filter
search_web(query="elections", search_type="news", region="us-en", provider="exa")

# Date-filtered search
search_web(query="uv package manager", time_range="w", provider="auto")

# Deep read a page
fetch_page(url="https://docs.python.org/3/library/os.html")

Technical Analysis

# Track GitHub issues/PRs
search_github(query="uv package manager")

# Get full GitHub issue thread
get_github_issue(url="https://github.com/astral-sh/uv/issues/1")

Community Sentiment

# Reddit discussions
search_reddit(query="Best mechanical keyboards 2024", subreddits=["MechanicalKeyboards"])

# Hacker News technical discourse
search_hackernews(query="MCP server architecture")

# LinkedIn professional search
search_linkedin(query="site reliability engineer", content_type="people")
search_linkedin(query="machine learning startup", content_type="companies")
search_linkedin(query="kubernetes devops", content_type="jobs")
search_linkedin(query="AI agents", content_type="posts")

Academic Research

# arXiv paper search with field prefixes
search_arxiv(query="au:Goodfellow AND cat:cs.LG")
search_arxiv(query="transformer attention", sort_by="submitted_date")

# Wikipedia background research
search_wikipedia(query="Quantum computing")

AI Trends & Forecasting

# Digg AI 1000 - curated clusters from high-signal X accounts
search_digg(query="AI agents")
search_digg(query="Claude Code", depth="deep")

# Techmeme - editorial tech news
search_techmeme(query="Apple WWDC")
search_techmeme(query="AI funding", depth="deep")

# Polymarket - prediction markets with real-money odds
search_polymarket(query="US election 2024")
search_polymarket(query="Fed rate cut")
search_polymarket(query="AI breakthrough", depth="deep")  # expanded query search

Result Refinement

# Rerank search snippets to top 3 most relevant (default: token_overlap, zero-dep)
rerank_results(
    query="how to cancel asyncio tasks",
    documents=["snippet1...", "snippet2...", "snippet3..."],
    top_n=3,
)

# Neural reranking with FlashRank (uv pip install 'web_search_mcp[flashrank]')
rerank_results(query="python async", documents=["..."], backend="flashrank")

# Hybrid scoring with metadata boosts and deduplication
rerank_results(
    query="kubernetes networking",
    documents=["..."],
    backend="hybrid",
    documents_meta=[{"engagement": 500, "freshness_days": 3, "authority": 0.9, "parent_id": "doc1"}],
    top_n=5,
)

---

🏗️ Project Structure

web_search_mcp/
├── server.py              # Entry point: FastMCP init, @mcp.tool registrations
├── search/                # Search engine implementations
│   ├── ddg.py             # DuckDuckGo search + trafilatura page fetch
│   └── exa.py             # Exa SDK search & content fetch (lazy-init client)
├── social/                # Community platform integrations
│   ├── github.py          # GitHub Search API + gh CLI issue rendering
│   ├── hackernews.py      # Algolia HN API + comment enrichment
│   ├── linkedin/          # LinkedIn search via DDG + Jina Reader
│   │   ├── __init__.py    # LinkedIn search tool registration
│   │   └── client.py      # DDG search + Jina Reader enrichment
│   ├── reddit/        # RSS + Shreddit keyless pipeline
│   │   ├── client.py      # HTTP client with RSS parsing
│   │   ├── parsers.py     # RSS/HTML parsers
│   │   ├── models.py      # Shreddit card parsing
│   │   ├── engine.py      # Search pipeline + arctic-shift backfill
│   │   ├── arctic_shift.py # Arctic-Shift score enrichment
│   │   └── _utils.py      # Shared HTML/regex helpers
│   └── x.py               # X/Twitter search via Xquik API or vendored Bird CLI
├── tools/                 # Specialized reference utilities
│   ├── arxiv.py           # arXiv paper search (Lucene field prefixes)
│   ├── wikipedia.py       # Wikipedia MediaWiki API
│   ├── digg.py            # Digg AI 1000 clusters via digg-pp-cli
│   ├── techmeme.py        # Techmeme tech news via techmeme-pp-cli
│   ├── polymarket.py      # Polymarket prediction markets via Gamma API
│   └── rerank.py          # Rerank documents against a query (token_overlap / flashrank / hybrid)
├── _config/               # Settings, env vars, rate limits, depth tiers
│   ├── settings.py        # pydantic-settings (EXA_API_KEY, SEARCH_MCP_ prefix)
│   └── limits.py          # Per-platform quick/default/deep limits, timeouts
├── _http/                 # Shared HTTP + SSRF protection
│   └── client.py          # validate_url, http_client, get_json_client
├── _models/               # Pydantic request/response models
│   ├── requests.py        # SearchRequest
│   ├── responses.py       # ErrorResponse, SearchResponse, PageResponse
│   └── types.py           # Depth, ResponseFormat, SearchType, FetchOutputFormat
├── _utils/                # Shared helpers
│   ├── dates.py           # Time-range and date-window utilities
│   ├── formatting.py      # Markdown formatters, date/epoch utils
│   ├── rate_limiter.py    # Token-bucket rate limiter
│   └── scoring.py         # Relevance scoring
└── vendor/                # Vendored third-party tools
    └── bird-search/       # Node.js CLI for X/Twitter search (fallback when XQUIK_API_KEY unset)

---

🛠️ Tool Implementation Flow

When adding a new tool:

1. Implement logic in the appropriate module (search/, social/, or tools/)
2. Define models in _models/ (request/response types)
3. Register in server.py using @mcp.tool decorator with a clear docstring (serves as the tool's description for the LLM)

---

📐 Design Decisions

• search-backend-split — Why search_web unifies DuckDuckGo and Exa behind a single provider parameter instead of exposing two separate tools.
• rerank-tool — Why rerank_results uses three scoring backends (token_overlap/flashrank/hybrid) with metadata boosts, dedup, and chunk diversity.

---

🧪 Testing

# Run all tests
uv run pytest

# Run a single test file
uv run pytest tests/test_module.py

# Run a specific test
uv run pytest tests/test_module.py::test_function_name

# Run with coverage
uv run pytest --cov=web_search_mcp

---

🔧 Troubleshooting

Problem	Likely Cause	Solution
Auth errors on a tool	Env var not set in the server's shell	Export the variable in the same shell where the MCP server process runs
GitHub returns empty results	Not authenticated	Run gh auth login or set GITHUB_TOKEN
search_x returns 401	Expired X session cookies	Re-extract auth_token and ct0 from x.com
fetch_page blocked by Cloudflare	Bot detection	Exa fallback is automatic; ensure EXA_API_KEY is set for server-side rendering
search_arxiv returns 503	Upstream arXiv maintenance	Wait a few minutes and retry
Tool says "Query cannot be empty"	Missing or blank query	Provide a non-empty search query

---

🤝 Contributing

1. Fork the repository.
2. Create a feature branch: git checkout -b feat/my-new-tool
3. Ensure all tests pass: uv run pytest
4. Submit a pull request with a detailed description of the changes.

---

📄 License

This project is licensed under the MIT License.