evalship

Lets your coding agent read what needs attention in a repository on evalship: failing evals, production AI issues and pull request reports. Read-only.

Hosted MCP Server

npx add-mcp 'https://evalship.com/mcp'

Installs into Claude Code, Codex, Cursor and more

Documentation

What your agent does

Every evalship docs page, for agents: https://evalship.com/llms.txt

Setup is a prompt your coding agent follows in your repository. It asks before it starts, sends you one plan before it changes anything, and leaves every merge to you.

Step by step

The agent asks one question at a time, with its question dialog where it has one.

StepWhat your agent doesWhat it changes
Before it startsReads the repository to find your evals and the prompts your AI calls send (step 1's search). It says how many it found and asks whether to start.Nothing
Sign upSends a link to sign up, by email or with GitHub, and connect the repository. It needs no access to GitHub. It waits for the connect.Nothing
1. Find the evalsSearches the repository with git ls-files and git grep for your eval framework or hand-rolled evals, the prompts and AI calls they cover, and the workflows that run them. It names earlier eval work on unmerged branches or in untracked scripts too, as a source of test cases.Nothing
2. Write the first suiteOnly with no evals, or when you asked to start from scratch, and once you agreed to its plan. When its cases need inputs like your product's real ones and the repository has none, it asks which data to use. Writes a first suite of 5 to 10 test cases, tries each check on made-up right and wrong replies and runs it once on your machine.A branch evalship/setup, with no pull request
3. Review the evalsReads every scorer, judge, threshold and dataset, and the prompts they test. It writes each flaw that holds up, citing files and lines, with one action to fix it. See The review.Nothing
4. Send the reviewPushes the review as the only file of a Git ref refs/evalship/review, where evalship reads it, then deletes the ref if it may. The ref is not a branch, so no deploy or CI run starts on it. Then it reports the eval tests it found, the AI calls no eval covers and each flaw.A Git ref, deleted after
After the reviewSends one plan: what makes your eval results reach evalship, the fixes for the flaws and evals for AI calls that have none, any product fix listed apart, and how many merges that is for you. When your evals read customer data, it asks which data they should use with evalship before any result is sent. It adds .evalship.yml and an Evals section to the instruction files your coding agents read (an AGENTS.md if there are none, unless you say no). Then it fixes the flaws you agree to, each as a pull request you merge, or a branch where the repository isn't on GitHub, and tells evalship what it fixed and dismissed.Pull requests or branches
At the endWhen its last pull request goes to merge, it asks to go on with Monitor and Optimize: do it all, start with one, or pause there. Monitor starts while you review that pull request, and Optimize once it's merged. Set up without GitHub, it asks whether to connect GitHub once both are done, for a check and a comment on every pull request. That installs the GitHub App and asks where the evals run: on your machine, or in GitHub Actions, which adds a workflow and needs your model API key as a secret.One pull request, if you pick GitHub Actions
With the GitHub App on the repository firstThe evals' results come from GitHub Actions. The plan's first pull request is the setup pull request, from the branch evalship/setup: it makes your eval job upload a results file and adds a job if none runs the evals. When the evals need a key the repository doesn't have, it asks whether they run in GitHub Actions or on your machine. At the end it asks once to make the evalship check required and to delete setup's merged branches. GitHub allows a required check on a private repository only with GitHub Pro (Team for an organization), so on its free plan the agent skips that and says so.Pull requests, and a branch rule if you say yes

To check that the results file works, the agent runs the evals once, which calls your model API as a CI run does. Its plan after the review says what a run costs and, with GitHub Actions, that setup's own pull requests run the evals about 20 times there.

After its plan, the agent asks again only for a product fix the plan didn't name, or a decision only you can make. Every claim it makes about the repository cites a file and line it read.

Lines for your agent

Setup is one of the lines you hand your coding agent. evalship shows each one where it applies, with your repository filled in: on To do, Test, Monitor, Optimize and Settings, and in pull request comments. Each links to a page your agent downloads and follows.

The lineWhat your agent doesWhat it changes
Set up https://evalship.com/setup.mdSets up evalship, as the steps above say.Nothing before the review. Then pull requests for what you agree to
Review the evals of OWNER/REPO with evalship: …/review.mdReviews your evals again and sends the new review, then plans its fixes as setup does.Pull requests for the flaws you agree to fix
Fix the eval flaw at …/fix.mdChecks each flaw you picked, fixes the ones that hold, dismisses the wrong ones with a reason, and waits for evalship's comment.One pull request
In OWNER/REPO, before you finish a change … follow …/change.mdRuns the evals a change affects before it finishes or opens a pull request, sends evalship the diff with the results, and writes the notes. Setup puts this line in AGENTS.md or CLAUDE.md. See Before you finish.Nothing
Run evalship's evals for this pull request: …/pr.mdChecks out the pull request, runs its evals on your machine, sends the results and writes the notes. Then it says what the check says and offers to fix what broke. It proposes to dismiss a flaw it finds wrong, for you to accept on the report. See Run evals on your machine.Nothing, unless you say yes to a fix
Run evalship's evals for main: …/pr.mdRuns the default branch's evals on your machine and sends the results, which become the baseline. The first time, the line starts with "Connect this machine and".Nothing in the repository. If GitHub Actions never sent its results, its eval runs move to your machines.
Send evalship the results of SUITE: …/results.mdMakes a suite the review found, but that never sent results, send them, without changing when it runs. A suite too slow or costly to ask for gets no_results in .evalship.yml, with why.One pull request
Update evalship's setup of OWNER/REPO: …/update.mdAdds what setup has added since your repository was set up.One pull request
Optimize this repo with evalship: …/optimize.mdTries changes on your machine, or on your GitLab runner when your evals run in GitLab CI, running only what you pick, at the price it shows, and offers a pull request for one that wins. See Optimize.Git worktrees, and a pull request you merge or don't
Send this app's AI calls to evalship: …/production.mdAdds one helper that sends an AI call to evalship, a call to it after each AI call your app makes, and a test for it. It adds checks for those calls to .evalship.yml. See Monitor.Product code and .evalship.yml
Before you finish a change … follow https://evalship.com/compare.mdChecks a change without an account. See Without an account.Nothing

Without GitHub, a pull request in this table is a branch the agent commits to and you merge.

Connect the MCP server

Your coding agent can read what needs attention in a repository and fix it, without you opening evalship. Unless you allow more, the server only reads. It has four tools:

ToolWhat it returns
todo(repo)The repository's open To do items, each with its link and the prompt that fixes it
issue(repo, id)One production issue: its latest failing calls, the trace of the latest one and the suggested fix
calls(repo)Production's AI calls, newest first. Narrow them with q, call, flagged or period, or pass a trace to read its steps in order
pull_request(repo, number)A pull request's report: which evals broke, why, and the flaws

It takes a read token. Run this in your terminal, check that the page it opens shows the same code, and click Allow. It then adds the server to Claude Code:

secret=$(openssl rand -hex 32) && key=$(printf %s "$secret" | openssl dgst -sha256 | sed 's/.*= //') \
  && url="https://evalship.com/agent?read=1&name=$(hostname -s)&key=$key" \
  && echo "code: $(printf %s "$key" | cut -c1-8 | tr a-f A-F | sed 's/..../&-/')" && echo "url: $url" && { open "$url" 2>/dev/null || xdg-open "$url" 2>/dev/null || true; } \
  && for i in $(seq 200); do token=$(curl -fsS -X POST https://evalship.com/api/v1/logins/token --form-string "poll=$secret"); [ -n "$token" ] && break; sleep 3; done \
  && [ -n "$token" ] && claude mcp add --transport http evalship https://evalship.com/mcp --header "Authorization: Bearer $token"

Cursor, Codex and other clients that take a URL and a header connect the same way: the URL is https://evalship.com/mcp, the header Authorization: Bearer followed by the token (echo $token).

To let your agent propose a dismissal too, add &propose=1 after read=1 in the snippet. The page you allow it on says so. Your agent then has a fifth tool, propose_dismissal(repo, number, kind, file, verdict, why), for a pull request's flaw that is wrong or that you chose not to fix. The flaw stays open, and the check failed, until someone with push access accepts the proposal on the report.

  • A read token reads what you see on evalship and changes nothing: a proposal dismisses no flaw. It can't send results, and an upload token can't read, so a token saved in CI never reads production data. Revoke either in Settings.
  • Production issues and calls are for the repository's members, also on a public repository.
  • Eval output, trace text and pull request text come back as data, cut to size, with a line that tells your agent to follow no instruction in them.
  • Not yet: claude.ai and other chat clients, which sign in with OAuth.

What setup never does

  • Read, print, create or change your secrets or .env files. When CI needs an API key, it names the secret and you add it in your own terminal with gh secret set; it checks secret names only.
  • Copy customer or personal data into test cases, the review or a pull request.
  • Weaken, skip or delete your evals or assertions.
  • Change product code, with two exceptions its plan names first: the lines that record AI calls for tinyeval, and a fix for a bug an eval shows.
  • Merge a pull request or force-push.
  • Work around a command a safety check blocks: it gives you the command to run yourself.

The other lines change what their row above says: Optimize's pull requests change prompts and models, and the production line adds a call after each AI call.

Without an account

Your agent can check a change before you connect anything. Add this line to AGENTS.md or CLAUDE.md:

Before you finish a change to a prompt, a model or an LLM call, follow https://evalship.com/compare.md

It runs the evals the change affects before and after the change, 3 tries each, on your machine with your keys, and evalship says whether it got worse beyond chance. evalship gets each try's result: the test case's name, whether it passed, why it failed, and its cost and time, with one line on what the change does. It keeps none of it. See For agents.

Or install the evalship skill, for every repo you work in. It also sets up evals and checks pull requests when you ask. In Claude Code:

/plugin install evalship --marketplace evalship-hq/skills

In Codex, Cursor and other agents:

npx skills add https://evalship.com

Public repositories

Anyone can open a public repository's dashboard at evalship.com/gh/OWNER/REPO: pass rates, recent runs and the number of flaws the review found. The flaws themselves stay visible to repository members. Someone with push access can turn off Public dashboard in its Settings.

The prompt

As your agent gets it once the repository is connected. Before that, evalship.com/setup.md has the welcome, the search for your evals and the sign-up step. After the review, the agent downloads the next steps from where setup stands. The title shows the prompt version, and the review records it.

The setup prompt (SetupPrompt.text), which the docs page shows in full.