Unified Docs Hub

Creates a massive, searchable knowledge base from numerous curated and auto-discovered GitHub projects.

GitHub

Verified on MseeP

🚀 Unified Docs Hub - The Ultimate MCP Documentation Server

License: MIT Python 3.8+ MCP Server

Transform your AI assistant into a documentation powerhouse! Unified Docs Hub is an MCP (Model Context Protocol) server that creates a massive, searchable knowledge base from 170+ curated repositories and 1000+ auto-discovered GitHub projects.

🌟 Why Unified Docs Hub?

Ever wished your AI assistant had instant access to ALL the documentation it needs? This MCP server solves that by:

  • 📚 Massive Knowledge Base: 170+ hand-picked repositories + 1000+ auto-discovered popular projects
  • 🔍 Lightning-Fast Search: Full-text search across 11,000+ documentation files in milliseconds
  • 🤖 AI-Optimized: Perfect for Claude, ChatGPT, and other AI assistants using MCP
  • 📈 Self-Updating: Automated daily updates and weekly discovery of new repositories
  • 🎯 Specialized Coverage: Deep expertise in Trading/Finance, AI/ML, DevOps, and 20+ categories

🎬 Real-World Examples

Example 1: Building a Trading Bot

AI: "Show me how to build a crypto trading bot with backtesting"

You: unified_search(query="crypto trading bot backtesting", category="Trading & Finance")

Result: Instant access to documentation from:
- Freqtrade (advanced crypto trading bot)
- Backtrader (backtesting framework)
- CCXT (100+ exchange APIs)
- TA-Lib (200+ technical indicators)

Example 2: Learning Kubernetes

AI: "Explain Kubernetes deployment strategies"

You: unified_search(query="kubernetes deployment strategies", category="Cloud/DevOps")

Result: Documentation from:
- Official Kubernetes docs
- Helm charts best practices
- ArgoCD GitOps workflows
- Istio service mesh patterns

Example 3: Machine Learning Pipeline

AI: "Set up an MLOps pipeline with experiment tracking"

You: unified_search(query="mlops pipeline experiment tracking", category="MLOps")

Result: Comprehensive guides from:
- MLflow (experiment tracking)
- Kubeflow (distributed training)
- DVC (data versioning)
- Weights & Biases (visualization)

📊 What's Inside?

Knowledge Coverage

CategoryRepositoriesHighlights
Trading & Finance64 reposAlgorithmic trading, options, forex, HFT, portfolio optimization
AI/ML20 reposLLMs, transformers, deep learning, NLP, computer vision
Cloud/DevOps15 reposKubernetes, Docker, Terraform, CI/CD, monitoring
Web Development12 reposReact, Vue, Next.js, full-stack frameworks
MLOps6 reposML lifecycle, experiment tracking, model deployment
Data Engineering8 reposApache Spark, Airflow, dbt, data pipelines
Observability5 reposPrometheus, Grafana, OpenTelemetry, APM
Blockchain5 reposSmart contracts, DeFi, Web3 development
20+ More Categories...Security, databases, mobile, desktop, and more

Key Features

  • 🔥 Full-Text Search: SQLite FTS5 engine for sub-second searches across millions of lines
  • 📈 Quality Scoring: Curated repos ranked by documentation quality (1-10 scale)
  • 🏷️ Smart Categorization: Browse by technology area or programming language
  • 🔄 Auto-Discovery: Continuously finds new popular repositories (10k+ stars)
  • 💾 Efficient Storage: Deduplication and compression keep the database lean
  • 🛡️ Rate Limit Handling: Respects GitHub API limits with smart throttling

🚀 Quick Start

Prerequisites

  • Python 3.8 or higher
  • GitHub Personal Access Token (optional but recommended)
  • An MCP-compatible AI assistant (Claude Desktop, Continue.dev, etc.)

Installation

  1. Clone the repository
git clone https://github.com/yourusername/unified-docs-hub.git
cd unified-docs-hub
  1. Set up Python environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
  1. Configure your MCP client

For Claude Desktop, add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "unified-docs-hub": {
      "command": "/path/to/unified-docs-hub/venv/bin/python",
      "args": ["/path/to/unified-docs-hub/unified_docs_hub_server.py"],
      "env": {
        "GITHUB_TOKEN": "your-github-token-here"
      }
    }
  }
}
  1. Initial indexing (optional - the server will do this automatically)
# Index all curated repositories
python -c "import asyncio; from unified_docs_hub_server import index_repositories; asyncio.run(index_repositories('smart'))"

📋 Available MCP Tools

unified_search

Search across all documentation with powerful filters.

# Basic search
unified_search("react hooks tutorial")

# Advanced search with filters
unified_search(
    query="transformer architecture attention",
    category="AI/ML",
    min_stars=5000
)

# Trading-specific search
unified_search(
    query="options greeks volatility smile",
    category="Trading & Finance"
)

index_repositories

Control repository indexing and discovery.

# Smart mode: Index curated + discover popular (recommended)
index_repositories(mode="smart")

# Update all existing repos
index_repositories(mode="update")

# Discover new trending repos
index_repositories(mode="discover", min_stars=5000, count=50)

list_repositories

Browse indexed repositories.

# List all Trading & Finance repos
list_repositories(category="Trading & Finance")

# Show only curated high-quality repos
list_repositories(source="curated", limit=20)

get_repository_docs

Get all documentation for a specific repository.

# Get all Kubernetes docs
get_repository_docs("kubernetes/kubernetes")

# Get trading library docs
get_repository_docs("freqtrade/freqtrade")

get_statistics

View comprehensive database statistics.

get_statistics()
# Returns: Total repos, documents, categories, languages, API status

🤖 Automated Updates

The server includes automated indexing that keeps your knowledge base fresh:

Setup Automated Updates

# Run the setup script
./setup_automated_indexing.sh

# Or manually start the updater
python automated_index_updater.py --once  # Run once
python automated_index_updater.py          # Run continuously

Update Schedule

  • Daily: Updates all curated repositories (2 AM, 2 PM)
  • Weekly: Discovers new trending repositories
  • On-Demand: Manual updates via MCP tools

🏗️ Architecture

Core Components

unified-docs-hub/
├── unified_docs_hub_server.py  # Main MCP server
├── database.py                 # SQLite + FTS5 engine
├── github_client.py            # GitHub API integration
├── response_limiter.py         # HTTP/2 error prevention
├── repositories.yaml           # Curated repo list
├── automated_index_updater.py  # Auto-update system
└── unified_docs.db            # Documentation database

How It Works

  1. Curation: Hand-picked repositories in repositories.yaml with quality scores
  2. Discovery: Automatically finds popular repos (10k+ stars) via GitHub API
  3. Indexing: Downloads and indexes README, docs/, and documentation files
  4. Storage: SQLite with FTS5 for efficient full-text search
  5. Serving: FastMCP server provides tools for AI assistants
  6. Updates: Automated system keeps documentation current

🎯 Use Cases

For AI Developers

  • Instant access to ML framework documentation
  • Compare different approaches across libraries
  • Find code examples and best practices

For Traders & Quants

  • Complete algorithmic trading documentation
  • Options pricing models and strategies
  • Backtesting frameworks and market data APIs

For DevOps Engineers

  • Kubernetes patterns and anti-patterns
  • CI/CD pipeline examples
  • Infrastructure as Code templates

For Full-Stack Developers

  • Frontend framework comparisons
  • Backend architecture patterns
  • Database optimization techniques

🛠️ Customization

Adding Custom Repositories

Edit repositories.yaml:

curated_repositories:
  - repo: "owner/awesome-project"
    category: "Web Development"
    description: "An awesome web framework"
    quality_score: 9
    priority: high
    doc_paths:
      - "docs/"
      - "README.md"
    topics: ["web", "framework", "javascript"]

Creating Custom Categories

Add new categories to group related technologies:

  - repo: "quantum-computing/qiskit"
    category: "Quantum Computing"  # New category!
    description: "Quantum computing SDK"

📈 Expansion Reports

See our journey of building this massive knowledge base:

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Ways to Contribute

  • Add high-quality repositories to repositories.yaml
  • Improve search algorithms
  • Add new MCP tools
  • Enhance documentation
  • Report bugs or request features

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Model Context Protocol for enabling AI-assistant integrations
  • All the amazing open-source projects indexed in our knowledge base
  • The GitHub API for making documentation discovery possible

📬 Contact

For questions, suggestions, or collaboration opportunities:

  • Open an issue on GitHub
  • Submit a pull request
  • Star the repository to show support!

Built with ❤️ for developers who want their AI assistants to know everything!

関連サーバー