Multi-LLM-MCP

あらゆるIDEエージェントが、クラウドAPIやローカルモデルなど任意のLLMにコーディングタスクを委任できるようにするMCPサーバー。単一の統一インターフェースを通じて実現します。

ドキュメント

multi-llm-mcp

MCP server + zero-dependency agent skill for multi-model second opinions.

The sharp angle is not "any model, any task" — it is a multi-model review tool: the same diff goes to 2–3 independent models in parallel, and what comes back is one merged, deduplicated critique where every finding is labeled with which models raised it.

  • Agreement (2+ models raise the same issue) → high-confidence, fix it.
  • Disagreement (only one model raises it) → exactly where a second opinion extends coverage a single model would have missed.

The tool assembles context itself (file paths or git diff HEAD) so the calling agent never pastes code, enforces a policy layer deciding what may leave the machine and which providers are allowed per repo, and reports tokens, latency, and estimated cost for every call.

Does a second model actually catch bugs the first one missed? Yes — measured. See the benchmark: on 18 functions each with one planted bug (two difficulty tiers, graded by an independent model), the realistic pairing — a small fast reviewer plus a reasoning reviewer — caught 18/18 merged vs 17/18 for the small model alone, and every finding agreed by both models was real (7/7), while all 6 hallucinated findings appeared only in single-model findings. Reproduce it yourself in two commands.


Features

  • review_diff — same context to N reviewers in parallel; merged, deduplicated findings; agreement stats; per-reviewer usage/cost
  • Context assembly — pass file paths, an explicit diff, or nothing at all (the tool runs git diff HEAD itself). Binary files, byte budgets, and untracked files are handled. The calling agent never pastes code.
  • Policy layer — .multi-llm-policy.json per repo: which paths may leave the machine, which providers are allowed, per-provider fallback chains, byte budgets. Secrets inside permitted paths are always redacted.
  • Fallback chains + timeouts + retries — providers=["groq", "deepseek"] tries in order; transient failures (timeouts, 429/5xx) retry with backoff, non-transient ones (404 model, missing key) fall through to the next provider.
  • Token & cost reporting — every tool call reports input/output tokens, latency, and estimated USD cost (price table, MULTI_LLM_MODEL_PRICES override; unknown prices are reported as null, never invented).
  • 5 providers out of the box — NVIDIA NIM, OpenRouter, Groq, DeepSeek, Ollama (local)
  • Standalone skill CLI — multi-llm-skill/ mirrors everything in a zero-dependency script (ask_llm.py, review.py) that works in any agent harness, no MCP required

Supported Providers

ProviderTypeModels
NVIDIA NIMCloudLlama 3.x, Mistral, DeepSeek, Code Llama, etc.
OpenRouterCloudClaude, GPT-4, Gemini, 200+ models
GroqCloudLlama 3.x, Mixtral (ultra-fast)
DeepSeekCloudDeepSeek Chat / Coder
OllamaLocalLlama 3, CodeLlama, Mistral, Gemma, etc.

Quick Start

1. Clone and install

git clone https://github.com/arjunkr303/multi-llm-mcp.git
cd multi-llm-mcp

python -m venv venv
source venv/bin/activate    # On Windows: venv\Scripts\activate

pip install -e .

2. Configure API keys

cp .env.example .env

Add only the keys you have (each provider is optional):

NVIDIA_API_KEY=your_nvidia_key_here
OPENROUTER_API_KEY=sk-or-v1-...
GROQ_API_KEY=gsk_...
DEEPSEEK_API_KEY=sk-...      # Ollama needs none

3. Connect to your IDE

Claude Desktop (claude_desktop_config.json) or Cursor (.cursor/mcp.json):

{
  "mcpServers": {
    "multi-llm-gateway": {
      "command": "/path/to/multi-llm-mcp/venv/bin/python",
      "args": ["-m", "multi_llm_mcp"]
    }
  }
}

VS Code (.vscode/mcp.json) — note the top-level servers key:

{
  "servers": {
    "multi-llm-gateway": {
      "command": "/path/to/multi-llm-mcp/venv/bin/python",
      "args": ["-m", "multi_llm_mcp"]
    }
  }
}

Replace /path/to/multi-llm-mcp with the actual clone path.

4. Use it

Once connected, the IDE agent gets three tools. Say thing like:

Review my working-tree changes with two models and tell me
what they disagree on.

or

Run check_policy on src/ and .env to see what could leave the machine.

The tools

review_diff — multi-model review (the flagship)

review_diff(
    paths=[],                 # files to review (relative paths)
    diff=None,                # explicit diff; empty => runs \`git diff HEAD\` itself
    focus=None,               # e.g. "security" or "error handling"
    reviewers=None,           # ["groq@llama-3.3-70b-versatile", "deepseek@deepseek-chat"]
    include_working_tree_diff=true,
    include_full_files=false, # also attach full content of changed files
    timeout=120, max_tokens=2048, concurrency=4
)
  • Context: paths and/or diff; with neither it reviews the current working tree diff automatically. Binary files are skipped, everything is capped to the policy byte budget, and untracked files are listed for you (pass them as paths to include their contents).
  • Reviewers: provider@model labels. Distinct model weights produce a genuine second opinion; defaults pick two providers with present keys. Per-reviewer fallback chains come from the policy.
  • Merge: title-similarity clustering (deterministic, no extra LLM call) assigns a found_by list + reviewer_count to every merged finding.

Returns JSON: merged findings, raw per-reviewer replies, agreement stats, usage (tokens/latency/cost per reviewer and totals), context audit (what was included/blocked), policy summary, and any errors.

Tip: spend attention in this order — findings with reviewer_count >= 2, then unique findings of the stronger reviewer, and treat all single-model claims as hypotheses until you verify them in the code.

ask_llm — delegated single task

ask_llm(
    prompt,                  # question, code snippet, or task
    system_prompt="You are a helpful coding assistant.",
    providers=None,          # fallback chain; entry = "provider" or "provider@model"
    model=None,              # model for entries without '@model'
    temperature=0.0, max_tokens=4096, timeout=120, report_usage=true
)

The response comes back plus a usage footer like:

Usage: in=1841 out=326 tok, 2.14 s, ~$0.0012 (served by groq@llama-3.3-70b-versatile, requested nvidia@meta/llama-3.3-70b-instruct)

check_policy — dry-run the send policy

check_policy(paths=["src/api.py", ".env"], providers=["groq"])

No API call is made. Shows which paths are permitted (with the exact policy rule that decided it), which providers are allowed for this repo, the byte budget, and the secret-redaction settings. Run it whenever you're unsure what review_diff would transmit.


The policy layer —.multi-llm-policy.json

Place one per repo root; optional (built-in defaults deny secret stores, dependency dirs, and VCS internals). Example:

{
  "allowed_providers": ["groq", "deepseek", "ollama"],
  "default_providers": ["groq", "deepseek"],
  "provider_fallbacks": {
    "groq": ["deepseek@deepseek-chat", "nvidia@meta/llama-3.3-70b-instruct"]
  },
  "send_allow": ["src/**", "tests/**"],
  "send_deny": [".env", ".env.*", "**/*.pem", "**/id_rsa*", "**/vendor/**"],
  "max_bytes_per_request": 128000,
  "on_deny": "skip",
  "secret_redact": true
}
KeyMeaning
allowed_providersWhich providers may serve this repo's files at all (null = all)
default_providersDefault ask/review routing when the caller doesn't specify
provider_fallbacksFallback chain per provider; "provider" reuses the requested model, "provider@model" pins one (the required form in review_diff chains)
send_allow / send_denyGlob rules for any path assembled into a prompt; allowlist is checked after the denylist; ..-escapes always denied
max_bytes_per_requestTotal byte budget for a single call's context
on_denyskip excluded paths, or error to abort instead
secret_redactSecrets inside permitted paths are redacted true (blocking error when false); secret scanning itself is always on

Fallback semantics: a bare "deepseek" reuses the model the caller asked for; "deepseek@deepseek-chat" pins a specific model — needed where indirect calls (review) have to know exactly which model to ask for.

Policy resolution order: $MULTI_LLM_POLICY env override → nearest .multi-llm-policy.json walking up from cwd → ~/.config/multi-llm/policy.json → built-in defaults. An invalid policy file falls back to the safe defaults (it never silently disables a deny rule), and secret scanning is always on regardless of any policy.


Benchmark: does a second model catch what the first missed?

bench/run_benchmark.py measures it with real calls:

  • 18 functions, each with exactly one objectively-planted bug, in easy and hard tiers (bench/dataset.py)
  • two independent reviewer models review the same functions via the production run_review path (the code review_diff actually executes)
  • a third grader model (never one of the reviewers), given the planted bug definition, judges per reviewer whether that review identified it
  • a precision pass (bench/precision.py) grades every reported finding as real bug vs hallucinated/speculative — the recall-vs-precision balance a shotgun reviewer could otherwise fake
  • headline metrics: per-model recall by tier, bugs found by A-only / B-only / both / neither, union coverage, tokens, latency, and whether agreement filtered out false positives

Headline result (realistic pairing — small fast reviewer + reasoning reviewer): 18/18 merged vs 17/18 for the small model alone, the small model ran ~50x faster, and 7/7 agreed findings were real while all 6 graded false positives appeared only in single-model findings. Two reasoning-model reviewers saturate the dataset (both 18/18, 100% precision) — pair by capability difference, not similarity.

Full numbers, second run, grading evidence, and caveats: bench/RESULTS.md. Reproduce with your own keys:

python bench/run_benchmark.py \
  --reviewers "nvidia@mistralai/mistral-large-4-1t-a52b,nvidia@poolside/laguna-xs-2.1" \
  --grader "nvidia@stepfun-ai/step-5-preview"

Cost & usage reporting

Every tool call surfaces usage telemetry:

  • Tokens: input + output per provider call. Cost: estimated USD from a small price table (DEFAULT_PRICES in src/multi_llm_mcp/providers.py), overridable entirely via MULTI_LLM_MODEL_PRICES='{"provider/model": {"input": 0.05, "output": 0.08}}'. Unknown price ⇒ "cost_usd": null + "cost_complete": false — never a made-up number. Ollama (local) is $0.
  • Latency per reviewer/call is included in every usage report.
  • The review JSON aggregates totals and flags cost_complete so the agent can treat its running cost as either exact or partial.

Security

  • API keys come from environment variables only; .env is gitignored
  • The policy layer decides what leaves the machine (see above); check_policy lets the agent inspect that decision before sending
  • Secret scanning is always on; detected secrets are redacted before send
  • Prompts go only to the providers you choose; for code that must not leave the machine at all, allowlist ollama in your policy and use it
  • See SECURITY.md for the full threat model

Project Structure

multi-llm-mcp/
├── src/multi_llm_mcp/        # MCP server package
│   ├── __init__.py
│   ├── __main__.py           # entry point: python -m multi_llm_mcp
│   ├── server.py             # FastMCP tools: ask_llm, review_diff, check_policy
│   ├── providers.py          # provider registry, clients, chat wrapper, cost
│   ├── context.py            # path/diff context assembly
│   ├── policy.py             # .multi-llm-policy.json loader + secret scanning
│   └── review.py             # parallel reviewers, JSON findings, merge
├── multi-llm-skill/          # standalone agent skill (zero deps, no MCP)
│   ├── SKILL.md              # skill metadata + agent instructions
│   ├── scripts/ask_llm.py    # single-task delegation CLI
│   ├── scripts/review.py     # multi-model review CLI (mirrors review_diff)
│   └── README.md
├── bench/                    # second-opinion benchmark
│   ├── dataset.py            #   18 planted-bug functions (easy + hard tiers)
│   ├── run_benchmark.py      #   run orchestration (uses the production pipeline)
│   ├── precision.py          #   false-finding (precision) grading pass
│   ├── render.py             #   machine summary generator
│   ├── RESULTS.md            #   written results (both runs + analysis)
│   └── results*.json         #   raw per-snippet reviews, grading, precision
├── .multi-llm-policy.json    # this repo's own policy (self-hosted)
├── multi-llm-policy.example.json
├── tests/
├── .github/workflows/ci.yml
├── pyproject.toml
└── .env.example

Running Tests

pip install -e ".[dev]"
pytest

License

MIT — see LICENSE.