Multi-LLM-MCP
一個讓任何 IDE 代理程式將編碼任務委派給任何 LLM 的 MCP 伺服器:無論是雲端 API 或本機模型,皆可透過單一統一介面完成。
文件
multi-llm-mcp
MCP server + zero-dependency agent skill for multi-model second opinions.
The sharp angle is not "any model, any task" — it is a multi-model review tool: the same diff goes to 2–3 independent models in parallel, and what comes back is one merged, deduplicated critique where every finding is labeled with which models raised it.
- Agreement (2+ models raise the same issue) → high-confidence, fix it.
- Disagreement (only one model raises it) → exactly where a second opinion extends coverage a single model would have missed.
The tool assembles context itself (file paths or git diff HEAD) so the calling agent never pastes code, enforces a policy layer deciding what may leave the machine and which providers are allowed per repo, and reports tokens, latency, and estimated cost for every call.
Does a second model actually catch bugs the first one missed? Yes — measured. See the benchmark: on 18 functions each with one planted bug (two difficulty tiers, graded by an independent model), the realistic pairing — a small fast reviewer plus a reasoning reviewer — caught 18/18 merged vs 17/18 for the small model alone, and every finding agreed by both models was real (7/7), while all 6 hallucinated findings appeared only in single-model findings. Reproduce it yourself in two commands.
Features
review_diff— same context to N reviewers in parallel; merged, deduplicated findings; agreement stats; per-reviewer usage/cost- Context assembly — pass file paths, an explicit diff, or nothing at all (the tool runs
git diff HEADitself). Binary files, byte budgets, and untracked files are handled. The calling agent never pastes code. - Policy layer —
.multi-llm-policy.jsonper repo: which paths may leave the machine, which providers are allowed, per-provider fallback chains, byte budgets. Secrets inside permitted paths are always redacted. - Fallback chains + timeouts + retries —
providers=["groq", "deepseek"]tries in order; transient failures (timeouts, 429/5xx) retry with backoff, non-transient ones (404 model, missing key) fall through to the next provider. - Token & cost reporting — every tool call reports input/output tokens, latency, and estimated USD cost (price table,
MULTI_LLM_MODEL_PRICESoverride; unknown prices are reported asnull, never invented). - 5 providers out of the box — NVIDIA NIM, OpenRouter, Groq, DeepSeek, Ollama (local)
- Standalone skill CLI —
multi-llm-skill/mirrors everything in a zero-dependency script (ask_llm.py,review.py) that works in any agent harness, no MCP required
Supported Providers
| Provider | Type | Models |
|---|---|---|
| NVIDIA NIM | Cloud | Llama 3.x, Mistral, DeepSeek, Code Llama, etc. |
| OpenRouter | Cloud | Claude, GPT-4, Gemini, 200+ models |
| Groq | Cloud | Llama 3.x, Mixtral (ultra-fast) |
| DeepSeek | Cloud | DeepSeek Chat / Coder |
| Ollama | Local | Llama 3, CodeLlama, Mistral, Gemma, etc. |
Quick Start
1. Clone and install
git clone https://github.com/arjunkr303/multi-llm-mcp.git
cd multi-llm-mcp
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -e .
2. Configure API keys
cp .env.example .env
Add only the keys you have (each provider is optional):
NVIDIA_API_KEY=your_nvidia_key_here
OPENROUTER_API_KEY=sk-or-v1-...
GROQ_API_KEY=gsk_...
DEEPSEEK_API_KEY=sk-... # Ollama needs none
3. Connect to your IDE
Claude Desktop (claude_desktop_config.json) or Cursor (.cursor/mcp.json):
{
"mcpServers": {
"multi-llm-gateway": {
"command": "/path/to/multi-llm-mcp/venv/bin/python",
"args": ["-m", "multi_llm_mcp"]
}
}
}
VS Code (.vscode/mcp.json) — note the top-level servers key:
{
"servers": {
"multi-llm-gateway": {
"command": "/path/to/multi-llm-mcp/venv/bin/python",
"args": ["-m", "multi_llm_mcp"]
}
}
}
Replace
/path/to/multi-llm-mcpwith the actual clone path.
4. Use it
Once connected, the IDE agent gets three tools. Say thing like:
Review my working-tree changes with two models and tell me
what they disagree on.
or
Run check_policy on src/ and .env to see what could leave the machine.
The tools
review_diff — multi-model review (the flagship)
review_diff(
paths=[], # files to review (relative paths)
diff=None, # explicit diff; empty => runs \`git diff HEAD\` itself
focus=None, # e.g. "security" or "error handling"
reviewers=None, # ["groq@llama-3.3-70b-versatile", "deepseek@deepseek-chat"]
include_working_tree_diff=true,
include_full_files=false, # also attach full content of changed files
timeout=120, max_tokens=2048, concurrency=4
)
- Context:
pathsand/ordiff; with neither it reviews the current working tree diff automatically. Binary files are skipped, everything is capped to the policy byte budget, and untracked files are listed for you (pass them aspathsto include their contents). - Reviewers:
provider@modellabels. Distinct model weights produce a genuine second opinion; defaults pick two providers with present keys. Per-reviewer fallback chains come from the policy. - Merge: title-similarity clustering (deterministic, no extra LLM call) assigns a
found_bylist +reviewer_countto every merged finding.
Returns JSON: merged findings, raw per-reviewer replies, agreement stats, usage (tokens/latency/cost per reviewer and totals), context audit (what was included/blocked), policy summary, and any errors.
Tip: spend attention in this order — findings with
reviewer_count >= 2, then unique findings of the stronger reviewer, and treat all single-model claims as hypotheses until you verify them in the code.
ask_llm — delegated single task
ask_llm(
prompt, # question, code snippet, or task
system_prompt="You are a helpful coding assistant.",
providers=None, # fallback chain; entry = "provider" or "provider@model"
model=None, # model for entries without '@model'
temperature=0.0, max_tokens=4096, timeout=120, report_usage=true
)
The response comes back plus a usage footer like:
Usage: in=1841 out=326 tok, 2.14 s, ~$0.0012 (served by groq@llama-3.3-70b-versatile, requested nvidia@meta/llama-3.3-70b-instruct)
check_policy — dry-run the send policy
check_policy(paths=["src/api.py", ".env"], providers=["groq"])
No API call is made. Shows which paths are permitted (with the exact policy rule that decided it), which providers are allowed for this repo, the byte budget, and the secret-redaction settings. Run it whenever you're unsure what review_diff would transmit.
The policy layer —.multi-llm-policy.json
Place one per repo root; optional (built-in defaults deny secret stores, dependency dirs, and VCS internals). Example:
{
"allowed_providers": ["groq", "deepseek", "ollama"],
"default_providers": ["groq", "deepseek"],
"provider_fallbacks": {
"groq": ["deepseek@deepseek-chat", "nvidia@meta/llama-3.3-70b-instruct"]
},
"send_allow": ["src/**", "tests/**"],
"send_deny": [".env", ".env.*", "**/*.pem", "**/id_rsa*", "**/vendor/**"],
"max_bytes_per_request": 128000,
"on_deny": "skip",
"secret_redact": true
}
| Key | Meaning |
|---|---|
allowed_providers | Which providers may serve this repo's files at all (null = all) |
default_providers | Default ask/review routing when the caller doesn't specify |
provider_fallbacks | Fallback chain per provider; "provider" reuses the requested model, "provider@model" pins one (the required form in review_diff chains) |
send_allow / send_deny | Glob rules for any path assembled into a prompt; allowlist is checked after the denylist; ..-escapes always denied |
max_bytes_per_request | Total byte budget for a single call's context |
on_deny | skip excluded paths, or error to abort instead |
secret_redact | Secrets inside permitted paths are redacted true (blocking error when false); secret scanning itself is always on |
Fallback semantics: a bare "deepseek" reuses the model the caller asked for; "deepseek@deepseek-chat" pins a specific model — needed where indirect calls (review) have to know exactly which model to ask for.
Policy resolution order: $MULTI_LLM_POLICY env override → nearest .multi-llm-policy.json walking up from cwd → ~/.config/multi-llm/policy.json → built-in defaults. An invalid policy file falls back to the safe defaults (it never silently disables a deny rule), and secret scanning is always on regardless of any policy.
Benchmark: does a second model catch what the first missed?
bench/run_benchmark.py measures it with real calls:
- 18 functions, each with exactly one objectively-planted bug, in
easyandhardtiers (bench/dataset.py) - two independent reviewer models review the same functions via the production
run_reviewpath (the codereview_diffactually executes) - a third grader model (never one of the reviewers), given the planted bug definition, judges per reviewer whether that review identified it
- a precision pass (
bench/precision.py) grades every reported finding as real bug vs hallucinated/speculative — the recall-vs-precision balance a shotgun reviewer could otherwise fake - headline metrics: per-model recall by tier, bugs found by A-only / B-only / both / neither, union coverage, tokens, latency, and whether agreement filtered out false positives
Headline result (realistic pairing — small fast reviewer + reasoning reviewer): 18/18 merged vs 17/18 for the small model alone, the small model ran ~50x faster, and 7/7 agreed findings were real while all 6 graded false positives appeared only in single-model findings. Two reasoning-model reviewers saturate the dataset (both 18/18, 100% precision) — pair by capability difference, not similarity.
Full numbers, second run, grading evidence, and caveats: bench/RESULTS.md. Reproduce with your own keys:
python bench/run_benchmark.py \
--reviewers "nvidia@mistralai/mistral-large-4-1t-a52b,nvidia@poolside/laguna-xs-2.1" \
--grader "nvidia@stepfun-ai/step-5-preview"
Cost & usage reporting
Every tool call surfaces usage telemetry:
- Tokens: input + output per provider call. Cost: estimated USD from a small price table (
DEFAULT_PRICESinsrc/multi_llm_mcp/providers.py), overridable entirely viaMULTI_LLM_MODEL_PRICES='{"provider/model": {"input": 0.05, "output": 0.08}}'. Unknown price ⇒"cost_usd": null+"cost_complete": false— never a made-up number. Ollama (local) is $0. - Latency per reviewer/call is included in every usage report.
- The review JSON aggregates totals and flags
cost_completeso the agent can treat its running cost as either exact or partial.
Security
- API keys come from environment variables only;
.envis gitignored - The policy layer decides what leaves the machine (see above);
check_policylets the agent inspect that decision before sending - Secret scanning is always on; detected secrets are redacted before send
- Prompts go only to the providers you choose; for code that must not leave the machine at all, allowlist
ollamain your policy and use it - See SECURITY.md for the full threat model
Project Structure
multi-llm-mcp/
├── src/multi_llm_mcp/ # MCP server package
│ ├── __init__.py
│ ├── __main__.py # entry point: python -m multi_llm_mcp
│ ├── server.py # FastMCP tools: ask_llm, review_diff, check_policy
│ ├── providers.py # provider registry, clients, chat wrapper, cost
│ ├── context.py # path/diff context assembly
│ ├── policy.py # .multi-llm-policy.json loader + secret scanning
│ └── review.py # parallel reviewers, JSON findings, merge
├── multi-llm-skill/ # standalone agent skill (zero deps, no MCP)
│ ├── SKILL.md # skill metadata + agent instructions
│ ├── scripts/ask_llm.py # single-task delegation CLI
│ ├── scripts/review.py # multi-model review CLI (mirrors review_diff)
│ └── README.md
├── bench/ # second-opinion benchmark
│ ├── dataset.py # 18 planted-bug functions (easy + hard tiers)
│ ├── run_benchmark.py # run orchestration (uses the production pipeline)
│ ├── precision.py # false-finding (precision) grading pass
│ ├── render.py # machine summary generator
│ ├── RESULTS.md # written results (both runs + analysis)
│ └── results*.json # raw per-snippet reviews, grading, precision
├── .multi-llm-policy.json # this repo's own policy (self-hosted)
├── multi-llm-policy.example.json
├── tests/
├── .github/workflows/ci.yml
├── pyproject.toml
└── .env.example
Running Tests
pip install -e ".[dev]"
pytest
License
MIT — see LICENSE.