mcp-drill

mcp-drill

Fault injection and reliability testing for MCP servers and AI agents. Wrap any Model Context Protocol server in one command; inject timeouts, malformed JSON-RPC, truncated and corrupted-but-valid tool outputs; measure whether the server degrades cleanly — and whether your agent notices or silently acts on the garbage.

uvx mcp-drill wrap --faults timeout,corrupt -- npx -y @modelcontextprotocol/server-filesystem /tmp

MCP is JSON-RPC over stdio/SSE with bidirectional notifications, so ordinary HTTP fault-injectors and chaos tools don't fit. mcp-drill speaks MCP: it sits transparently between an MCP client and a backend server and perturbs the traffic, so you can test failure paths in CI without a live LLM.

Finding: across 31 popular MCP servers (including Microsoft Learn, Hugging Face, Cloudflare and DeepWiki), only 3% of tools declare an output contract that would reject a corrupted response. See the live scorecard.

Why

Real MCP deployments fail in ways integration tests never cover: a tool times out, a server returns a well-formed-but-wrong payload, a response is truncated mid-stream. Most agents were never exercised against these paths. mcp-drill makes them reproducible:

• 🧪 Fault injection — deterministically inject timeouts, malformed/oversized/truncated responses, corrupted-but-schema-valid payloads, dropped tools, and latency.
• 🎬 CI-first — a CLI and a GitHub Action; no live model or API keys required in the loop.
• 📊 Reliability scorecard — a model-free scan that grades how a server responds to bad input and how machine-checkable its tool output contracts are.

Install

pip install mcp-drill[scan]        # or: pipx install mcp-drill[scan]
uvx mcp-drill scan -- --help       # no install, run once
npm i -g mcp-drill                 # shim: prints version + points to PyPI

Quickstart

# wrap a server and inject faults into its responses
mcp-drill wrap --faults timeout,truncate -- npx -y @modelcontextprotocol/server-everything

# score a local (stdio) server's fault handling and output-schema hygiene (no LLM involved)
mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp

# score a remote server over Streamable HTTP (add --header for auth if needed)
mcp-drill scan --url https://mcp.deepwiki.com/mcp

# emit a shields.io badge for a server's output-contract grade
mcp-drill scan --badge --url https://mcp.deepwiki.com/mcp

What it measures (model-free)

The scan command is deterministic and involves no language model, so its numbers are properties of the server and the protocol — not of whichever agent happens to call it:

1. Error conformance — on invalid requests (unknown method, unknown tool, missing required arguments), does the server return a spec-compliant JSON-RPC error, a proper tool error, or does it hang / crash / answer as if nothing was wrong?
2. Output-contract coverage — what fraction of a server's tools declare a machine-checkable outputSchema at all? Tools without one give downstream validators nothing to check.
3. Output-contract enforceability — of the tools that declare an outputSchema, how many would actually reject a corrupted (well-typed but wrong) response? Many declared schemas validate shape only, so a corrupted payload still passes the server's own contract.

Status

Early development. The proxy/injector core is pure-stdlib; schema scoring uses jsonschema (the scan extra). Telemetry is off — the tool never phones home. See vs mcp-scan for how mcp-drill (contract compliance) differs from mcp-scan (security).

License

Apache-2.0 (see LICENSE).