LLMTest MCP Server

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

{
  "mcpServers": {
    "llmtest": {
      "command": "npx",
      "args": ["llmtest-mcp"],
      "env": {
        "LLMTEST_API_KEY": "llmt_your_key_here"
      }
    }
  }
}

3. Talk to your AI

Just ask in natural language:

• "Check my LLMTest status"
• "Find cheaper models for my AI calls"
• "Run a benchmark on my blog-writer flow"
• "What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

Tool	Description
status	Show proxy status and activity summary
list_flows	List all AI flows with cost and latency stats
get_suggestions	Get pending model-switch recommendations
update_suggestion	Accept or dismiss a suggestion
run_benchmark	Benchmark a flow against challenger models
optimize_prompt	Rewrite a flow's prompt and find a cheaper model that still works
seed_samples	Add test prompts for pre-launch benchmarking
list_samples	Show stored test samples per flow
list_new_models	Show new and trending models
get_account	Check credit balance and usage
get_autopilot_status	Check whether autopilot is on and whether the account is eligible
enable_autopilot	Turn on weekly auto-optimization with safety gates + drift-based auto-revert
disable_autopilot	Turn off autopilot (existing optimizations stay active)
list_active_optimizations	List auto-accepted optimizations still inside their 24h revert window
revert_optimization	Roll an auto-accepted optimization back to the previous prompt

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

• Autopilot enabled on the account
• Email verified
• Account age ≥ 14 days (trust ramp)
• Flow has ≥ 20 real calls in the last 7 days
• Flow not optimized by autopilot in the last 14 days (cooldown)
• Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

1. Tell your AI: "I'm building a support chatbot using gpt-4o"
2. It seeds realistic test samples with seed_samples
3. It runs run_benchmark to compare models
4. It shows you get_suggestions with cheaper alternatives

Post-launch (with real traffic):

1. Route your AI calls through https://llmtest.io/v1
2. LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
3. Ask "any cost-saving suggestions?" to see recommendations
4. Accept a suggestion and update your code

Environment Variables

Variable	Required	Description
LLMTEST_API_KEY	Yes	Your API key from llmtest.io/dashboard
LLMTEST_BASE_URL	No	Custom API URL (defaults to https://llmtest.io)

Links

• Website
• Dashboard
• Docs
• GitHub
• npm

License

MIT