conKurrence
AI evaluation toolkit — measure inter-rater agreement (Fleiss' κ, Kendall's W) across multiple LLM providers
ConKurrence
One command. Find out if your AI agrees with itself.
ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.
Install
npm install -g conkurrence
MCP Server
Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:
npx conkurrence mcp
Claude Desktop Configuration
Add to your claude_desktop_config.json:
{
"mcpServers": {
"conkurrence": {
"command": "npx",
"args": ["-y", "conkurrence", "mcp"]
}
}
}
Claude Code Plugin
/plugin marketplace add AlligatorC0der/conkurrence
Features
- Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously
- Statistical rigor — Fleiss' kappa with bootstrap confidence intervals, Kendall's W for validity
- Self-consistency mode — No API keys needed; uses the host model via MCP Sampling
- Schema suggestion — AI-powered schema design from your data
- Trend tracking — Compare runs over time, detect agreement degradation
- Cost estimation — Know the cost before running
MCP Tools
| Tool | Description |
|---|---|
conkurrence_run | Execute an evaluation across multiple AI raters |
conkurrence_report | Generate a detailed markdown report |
conkurrence_compare | Side-by-side comparison of two runs |
conkurrence_trend | Track agreement over multiple runs |
conkurrence_suggest | AI-powered schema suggestion from your data |
conkurrence_validate_schema | Validate a schema before running |
conkurrence_estimate | Estimate cost and token usage |
Links
- Homepage: conkurrence.com
- npm: npmjs.com/package/conkurrence
- Terms of Service: app.conkurrence.com/terms
- Privacy Policy: app.conkurrence.com/privacy
License
BUSL-1.1 — Business Source License 1.1
Servidores relacionados
Scout Monitoring MCP
patrocinadorPut performance and error data directly in the hands of your AI assistant.
Alpha Vantage MCP Server
patrocinadorAccess financial market data: realtime & historical stock, ETF, options, forex, crypto, commodities, fundamentals, technical indicators, & more
ComfyUI MCP Server
Integrates ComfyUI with MCP, allowing the use of custom workflows. Requires a running ComfyUI server.
Futu MCP
A quantitative analysis platform for Futu Securities, offering intelligent caching, technical analysis, and pattern recognition.
Remote MCP Server (Authless)
An example of a remote MCP server deployable on Cloudflare Workers, without authentication.
MCP Inspector
A developer tool for testing and debugging MCP servers with a web-based UI.
Remote MCP Server (Authless)
An example of a remote MCP server without authentication, deployable on Cloudflare Workers.
MCP Go Generator Node.js
Generate Go microservices with a hexagonal architecture in a Node.js environment.
SVGR
A MCP server that exposes SVGR functionality to convert SVG content into React components
OpenExp
Q-learning memory for Claude Code. Persistent memory that learns which context helps you get work done. Memories that lead to productive sessions (commits, PRs, tests) earn higher retrieval rank automatically. 16 MCP tools, hybrid BM25 + vector + Q-value scoring, local-first with Qdrant + FastEmbed.
Starwind UI
A server providing tools for developers working with Starwind UI components.
Claude Prompts MCP Server
A universal MCP server that loads prompts from an external JSON configuration file.