Iris

Iris — stop shipping agents on vibes

Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 13 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.

Requires Node.js 20 or later. Check with node --version.

A failure on screen in 60 seconds

No agent wiring, no config — one command:

npx @iris-eval/mcp-server --demo

This seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.

Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:

npx @iris-eval/mcp-server --demo-clear

Hook up your own agent

Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:

{
  "mcpServers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server", "--dashboard"]
    }
  }
}

Your agent discovers Iris's nine tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:

Log that last task to Iris and evaluate the output.

The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.

One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and SDKs on the roadmap will be thin clients over the same endpoint.

Capture over HTTP (no model in the loop)

With the dashboard running, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request:

curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "support-bot",
    "input": "What is the refund policy?",
    "output": "Refunds are available within 30 days of purchase.",
    "evaluate": true,
    "eval_type": "safety"
  }'

Returns 201 with the stored trace_id and the evaluation result. The endpoint accepts the same body as the log_trace tool and sits behind the same loopback-only middleware stack as the rest of the dashboard. Full contract, field reference, and error semantics: docs/http-ingest.md.

Check the install

npx @iris-eval/mcp-server --self-test

An offline install diagnostic: storage round-trip, deterministic evals, dashboard + DNS-rebinding guard — all inside an isolated temp home, so your real database is never opened. Exit code 0 = healthy, 1 = a check failed.

claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

{
  "servers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server"]
    }
  }
}

{
  "context_servers": {
    "iris-eval": {
      "command": {
        "path": "npx",
        "args": ["@iris-eval/mcp-server"]
      }
    }
  }
}

[mcp_servers.iris-eval]
command = "npx"
args = ["@iris-eval/mcp-server"]

Other Install Methods

# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard

# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint)
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-server

Tip: Global install (npm install -g) stores traces persistently at ~/.iris/iris.db. With npx, traces persist in the same location, but startup is slower due to package resolution.

What You Get

Where this is going next: the roadmap.

MCP Tools

Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:

• log_trace — Log an agent execution with spans, tool calls, token usage, and cost
• evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
• get_traces — Query stored traces with filtering, pagination, and time-range support
• list_rules — Enumerate deployed custom eval rules (read-only)
• deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
• delete_rule — Remove a deployed custom rule (destructive, idempotent)
• delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
• evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
• verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.

When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.

How passed is decided

evaluate_output returns both a score and a passed flag — they answer different questions:

• score (0..1) is the weighted average across the rules that ran — a quality gradient.
• passed is the ship/no-ship verdict: true only when the score clears the pass threshold (default 0.7) and no critical rule failed.

Genuine safety violations hard-fail. no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know: a critical rule that skipped (missing context, or any other cause of a skip) has not judged the output and does not veto — rule_results shows every skip and its reason, so a gate that must fail closed on non-verdicts can.

One gotcha for CI gates: if you omit eval_type, the default completeness bundle runs — safety rules don't. The response echoes eval_type (plus a note when it was defaulted) so your gate can verify which bundle actually ran. Key on passed for the verdict and eval_type: "safety" for coverage.

Full tool schemas and configuration: iris-eval.com

Hosted features

Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.

Hosted storage, shared team history and alerting are under consideration, not under construction. There is no pricing, and nothing to buy. If shared history would be useful to you, the waitlist is how we find out whether it's worth building — it commits you to nothing.

Two commitments hold regardless: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.

Examples

• Claude Desktop setup — MCP config for stdio and HTTP modes
• TypeScript — MCP SDK client — connect and invoke tools
• HTTP transport (TS + Python) — full client code for REST-style integration
• LangChain instrumentation (Python, conceptual) — scaffold showing the shape; needs your agent code to be runnable
• CrewAI instrumentation (Python, conceptual) — scaffold; same caveat

Community

• GitHub Issues — Bug reports and feature requests
• GitHub Discussions — Questions and ideas
• Contributing Guide — How to contribute
• HTTP Ingest — Deterministic trace capture via POST /api/v1/traces
• Roadmap — What's coming next

# Production deployment
iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboard

npx @iris-eval/mcp-server --self-test

npx --yes @iris-eval/mcp-server@latest

npm install -g @iris-eval/mcp-server@latest

npx @iris-eval/mcp-server --dashboard
# First log line: "Starting Iris MCP server vX.Y.Z"

# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest

# If installed globally
npm update -g @iris-eval/mcp-server

node --version  # Must be v20.x or v22.x+

# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server

# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"

If Iris is useful to you, consider starring the repo — it helps others find it.

MIT Licensed.