Web Scout
A server for web scraping, searching, and analysis using multiple engines and APIs.
Web-Scout: AI-Powered Search with LLM Summarization
A FastAPI application that performs web searches using DuckDuckGo and generates AI-powered summaries using Google's Gemini AI.
Features
- Web search using DuckDuckGo
- AI summarization using Gemini 2.5 Flash
- Two output modes: Summary and Detailed analysis
- Docker & Docker Compose ready
- Secure API key management via environment variables
- MCP (Model Context Protocol) support over HTTP
Prerequisites
- Docker and Docker Compose installed
- Google Gemini API key
Setup
-
Clone the repository (or navigate to your project directory)
-
Add your Gemini API key to the
.envfile:GEMINI_API_KEY=your_actual_gemini_api_key_here -
Build and run with Docker Compose:
docker-compose up --build
API Usage
The application will be available at http://localhost:8000
Health Check
curl http://localhost:8000/health
Search Endpoint
Summary Mode (Default)
curl "http://localhost:8000/search?query=artificial+intelligence"
# or explicitly specify mode=summary
curl "http://localhost:8000/search?query=artificial+intelligence&mode=summary"
Detailed Mode
curl "http://localhost:8000/search?query=artificial+intelligence&mode=detailed"
Response Format
{
"query": "your search query",
"mode": "summary",
"summary": "AI-generated analysis...",
"sources_used": 10
}
MCP Server Integration
Web-Scout can also function as an MCP (Model Context Protocol) server, allowing AI assistants to perform web searches directly through tools.
MCP Features
- Web Search Tool: Perform web searches with AI summarization
- Dual Mode Support: Both summary and detailed analysis modes
- HTTP Transport: MCP over HTTP protocol for client integration
- JSON Responses: Structured output for easy integration
MCP Tools Available
Web Search Tool
- Name:
web_search - Description: Perform a web search using DuckDuckGo and generate AI-powered summaries
- Parameters:
query(string, required): The search query to performmode(string, optional): Response mode - "summary" or "detailed" (default: "summary")
MCP Server Setup
Web-Scout provides MCP functionality over HTTP, accessible at the /mcp endpoint.
Method 1: Direct FastAPI Server
- Install dependencies:
pip install -r requirements.txt
- Set your Gemini API key:
export GEMINI_API_KEY=your_api_key_here
- Run the HTTP server with MCP endpoint:
python main.py
The MCP endpoint will be available at http://localhost:8000/mcp
Method 2: Docker Container
# Run the HTTP server with Docker (MCP available at /mcp)
docker-compose up --build
Or run standalone container
docker run -p 8000:8000 -e GEMINI_API_KEY=your_api_key_here web-scout
Integrating with AI Tools
To use Web-Scout as an MCP server with AI tools like Claude Desktop or Roo:
- Create MCP Configuration:
{
"mcpServers": {
"web-scout": {
"command": "python",
"args": ["-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"],
"env": {
"GEMINI_API_KEY": "your_gemini_api_key_here"
}
}
}
}
-
Configure your AI tool to use the MCP configuration:
- For Claude Desktop: Add to
~/Library/Application Support/Claude/claude_desktop_config.json - For Roo: Add to the appropriate configuration file
- For Claude Desktop: Add to
-
Usage Example:
Can you search for the latest news about artificial intelligence?
The AI tool will use the Web-Scout MCP server (via the /mcp endpoint) to perform the search and provide summarized results.
Development
Local Development (without Docker)
# Install dependencies
pip install -r requirements.txt
# Set your API key
export GEMINI_API_KEY=your_api_key_here
# Run the application
uvicorn main:app --reload
Using Docker Compose
# Build and run
docker-compose up --build
# Run in background
docker-compose up -d --build
# Stop the application
docker-compose down
# View logs
docker-compose logs -f web-scout
Configuration
Environment Variables
GEMINI_API_KEY: Your Google Gemini API key (required)
Docker Configuration
- Port: 8000
- Container name: web-scout
- Health check: Automatic health monitoring
Security Notes
- The
.envfile is ignored by Git and should never be committed - API keys are mounted securely via Docker Compose volumes
- The application uses health checks for monitoring
Error Handling
- Returns proper HTTP status codes
- Includes detailed error messages
- Handles missing API keys and invalid parameters gracefully
İlgili Sunucular
Bright Data
sponsorDiscover, extract, and interact with the web - one interface powering automated access across the public internet.
MCP RSS Crawler
Fetches and caches RSS feeds using a SQLite database for use with LLMs via the MCP protocol.
Career Site Jobs
A MCP server to retrieve up-to-date jobs from company career sites.
GitPrism
GitPrism is a fast, token-efficient, stateless pipeline that converts public GitHub repositories into LLM-ready Markdown.
Stepstone
Fetches job listings from Stepstone.de based on keywords and location parameters.
rssmcp
Fetches RSS feeds and returns entries as formatted text, with support for OPML configuration.
SearchMCP
Connect any LLM to the internet with the cheapest, most reliable, and developer-friendly search API.
transcriptor-mcp
An MCP server (stdio + HTTP/SSE) that fetches video transcripts/subtitles via yt-dlp, with pagination for large responses. Supports YouTube, Twitter/X, Instagram, TikTok, Twitch, Vimeo, Facebook, Bilibili, VK, Dailymotion. Whisper fallback — transcribes audio when subtitles are unavailable (local or OpenAI API). Works with Cursor and other MCP host
WebWatch MCP
Website change monitoring with URL watching, snapshot comparison, and intelligent content extraction for AI assistants
Website to Markdown MCP Server
Fetches and converts website content to Markdown with AI-powered cleanup, OpenAPI support, and stealth browsing.
Scrapfly
Scrapfly MCP Server gives AI agents a simple, unified way to scrape live web data with built-in anti-bot handling.