Scorable

For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs; this page is available as Markdown.

MCP Server

Scorable runs a hosted remote MCP server at https://api.scorable.ai/mcp. Point an MCP client at it with your API key and your agent gets 14 tools for finding, running, and authoring evaluations.

Use it when you want the agent to evaluate as part of its own reasoning loop: score a draft before showing it to the user, check a change against a rubric, or investigate why a score regressed. For scripted and CI use, the CLI is usually the better fit.

Get an API key

Grab one from app.scorable.ai under API keys, or mint a free temporary one from the terminal:

curl -s -X POST https://api.scorable.ai/create-demo-user/ | jq -r .api_key

Keep it in an environment variable so it never lands in a config file you might commit:

export SCORABLE_API_KEY="your-key"

Claude Code

claude mcp add --transport http scorable https://api.scorable.ai/mcp \
  --header "Authorization: Bearer $SCORABLE_API_KEY"

That registers the server for the current project. Add --scope user to make it available in every project, or --scope project to write it into .mcp.json and share it with your team through version control — in that case reference the variable rather than the key itself, so no secret is committed.

Verify it connected:

claude mcp list

Codex

Codex configures remote servers in ~/.codex/config.toml — codex mcp add is for stdio servers only:

[mcp_servers.scorable]
url = "https://api.scorable.ai/mcp"
bearer_token_env_var = "SCORABLE_API_KEY"

bearer_token_env_var reads the key from your environment at startup and sends it as a bearer token, so the key stays out of the file.

Cursor, VS Code, and other clients

Anything that speaks streamable HTTP MCP works. Most clients use this shape:

{
  "mcpServers": {
    "scorable": {
      "url": "https://api.scorable.ai/mcp",
      "headers": {
        "Authorization": "Bearer ${SCORABLE_API_KEY}"
      }
    }
  }
}

The server also accepts the Api-Key scheme used by the rest of the Scorable API, so Authorization: Api-Key <key> works if that fits your client better.

The tools

Finding what exists

Tool	What it does
list_judges	Judges available to your organization, newest first
get_judge	One judge in full, including every evaluator it applies
list_evaluators	Evaluators, including the presets Scorable ships
get_evaluator	One evaluator with its rubric and required inputs
list_projects	Projects, for scoping other calls

Running evaluations

Tool	What it does
run_judge	Score a request/response pair against a judge, by id or name
run_evaluator	Score against a single evaluator

Authoring

Tool	What it does
generate_judge	Build a judge from a plain-language description of what you care about
create_judge / update_judge	Create or edit a judge from an explicit evaluator list
create_evaluator / update_evaluator	Create or edit a single evaluator and its rubric

Auditing

Tool	What it does
list_execution_logs	Past runs, filterable by judge, project, score, cost, tags, and date
get_execution_log	One run in full, with per-evaluator scores and justifications

There are deliberately no delete tools. Removing a judge or evaluator stays a human action in the UI or CLI.

Try it

Once connected, prompts like these resolve to tool calls:

What Scorable judges do I have?

Generate a judge that checks our support replies are concise, grounded in the
policy documents, and never promise refunds the policy does not allow.

Run that judge against this reply: "Absolutely, I've processed a full refund!"
given the policy "Refunds within 30 days, unopened items only."

Why did last night's evaluation scores drop? Check the execution logs.

The generated judge in that third example returns a score per evaluator with a written justification, so the agent can act on why something failed rather than just a number.

Concepts worth knowing

An evaluator scores one quality of a response — faithfulness to context, relevance, safety, tone, or a custom rubric — returning a score between 0 and 1 with a justification. A judge is a reusable, named set of evaluators applied together, and is the normal unit of work.

The server tells your agent this on connect, so it generally picks the right tool without prompting. See Concepts for the full model.

Troubleshooting

401 Unauthorized — the key is missing, expired, or malformed. Confirm it works against the REST API first: curl -H "Authorization: Api-Key $SCORABLE_API_KEY" https://api.scorable.ai/v1/judges/?limit=1.

404 Not Found — the endpoint is disabled on that deployment. Self-hosted installations can turn it off; check that MCP_ENABLED is not set to false.

A tool reports a missing field — some evaluators require contexts or expected_output. Call get_judge and check each evaluator's requires_contexts and requires_expected_output before running.