ModelLens

ModelLens

See which of your models actually works.

An MCP server that turns any OpenAI-compatible gateway into an observable, rankable set of models. It discovers what a provider advertises, probes what works right now — latency, success, cancellation, streaming, tool calling — keeps the history, and answers "which model should I use?"

Zero third-party dependencies. Python 3.9+ standard library only.

---

Why

If you run more than one model provider, you already have this problem:

• The gateway says it serves 89 models. How many actually answer?
• Which one is fast today? Yesterday's answer is already stale.
• Your router picks the first entry in a list — and the first entry is slow.
• Failures are invisible until a user complains.

ModelLens makes that visible. It is the difference between trusting a model list and measuring it.

Real example from a live gateway (89 models, 16 probed):

magpie/kilo/nvidia/nemotron-3.5-super-vl:free     97.6  100.0%    0.0%      628ms   healthy
magpie/auto/beam-free                             96.6  100.0%    0.0%      909ms   healthy
magpie/longcat/LongCat-2.5                        88.4  100.0%    0.0%     3104ms   mixed
magpie/group/daily                                70.0  100.0%    0.0%    77345ms   mixed
magpie/radeon-cloud/DeepSeek-V4.1-Flash           70.0  100.0%    0.0%    87388ms   mixed
magpie/nvidia/deepseek-ai/deepseek-v4.1-flash     0.0    0.0%  100.0%         -    unreliable
magpie/modelscope/Qwen/Qwen3.8-27B                 0.0    0.0%    0.0%         -    quota_blocked

The default routing group was putting an 87-second model first and a 0.9-second model third. ModelLens found that in one pass.

---

Install

# from PyPI
pip install modellens-mcp
modellens --config config.json --data history.json --once ranking

or straight from source — no dependencies, no install step:

curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
python3 modellens.py --config config.json --data history.json --once ranking

Configure

{
  "endpoints": [
    {"name": "magpie",  "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
    {"name": "openai",  "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
    {"name": "ollama",  "base_url": "http://127.0.0.1:11434/v1"}
  ]
}

api_key may be empty for a localhost gateway. models optionally filters the advertised list; enabled: false disables an endpoint without deleting it.

Run as an MCP server

Point any MCP client at it:

{
  "mcpServers": {
    "modellens": {
      "command": "python3",
      "args": ["/path/to/modellens.py",
               "--config", "/path/to/config.json",
               "--data",   "/path/to/history.json"]
    }
  }
}

Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.

DeepSeek Harness

Register it in the profile's cordis.patch.yml (note: a new entry needs - insert:, not a bare - id: — a bare one means "modify an existing entry"):

- insert:
    - id: modellens
      name: "@deepseek-ai/dsh-mcp-client"
      config:
        serverName: "modellens"
        transport: "stdio"
        command: "python3"
        args:
          - "/data/workspace/modellens/modellens.py"
          - "--config"
          - "/data/workspace/modellens/config.json"
          - "--data"
          - "/data/workspace/modellens/history.json"

Then restart dsh web. Verified: the server runs as a child process of dsh-web (pgrep -P $(systemctl --user show dsh-web -p MainPID)).

Tools

Tool	What it does
list_models	What each endpoint advertises (optional substring filter)
probe	Real requests: latency, success, streaming, tool calls
ranking	Quality ranking from recorded history
recommend	Best model for a profile: fast | cheap | capable
report	The whole picture as markdown

Example

probe(limit=20, check_tools=true)
ranking(limit=10)
recommend(profile="fast")

---

How it scores

Every probe is a real request, so the numbers are evidence, not estimates.

Signal	Weight	Notes
Success rate	50%	0% success ⇒ score 0, regardless of anything else
Median latency	30%	Linear to 0 at 8 s
Cancellation rate	20%	Timeouts and client cancellations

Verdicts: healthy (≥95% success, <3 s median) · mixed · unreliable (≥30% cancelled) · quota_blocked (all 429) · dead (all 404) · untested.

History is kept per model (newest 200 probes) in history.json, so rankings improve as you use it.

---

CLI

The server doubles as a command-line tool:

python3 modellens.py --config config.json --data history.json --once probe --limit 20
python3 modellens.py --config config.json --data history.json --once ranking

---

Use cases

• Before you pick a model — recommend(profile="fast")
• When a provider goes quiet — ranking shows it dropping before your users do
• When your router misbehaves — compare group members, find the slow one
• Cost control — spot the expensive model that is never chosen
• Onboarding a new provider — probe once, see what actually works

---

Status and roadmap

Working today: discovery, probing, streaming/tool-call checks, scoring, ranking, recommendation, markdown report, MCP stdio transport, CLI.

Roadmap: OpenAI-compatible /v1/models only for now; Anthropic and Gemini transports, per-endpoint cost accounting, alerting webhooks, and a hosted multi-team version.

---

License

MIT. Use it, ship it, charge for your own hosted version.

---

Honest notes

• Probing costs tokens. Each probe is a real completion. Keep limit sensible and don't probe a hundred models every minute.
• A single probe is weak evidence. The score is meaningful after several probes; the report says n=1 so you can see the sample size.
• Latency depends on you. Your network, not just the provider, is in the measurement. Run it from where your agents run.