vitest-evals

por sentry

Use ao criar, revisar ou depurar suítes de vitest-evals com suporte de harness, adaptadores Harness personalizados, integrações de harness de primeira parte com ai-sdk ou pi-ai,…

npx skills add https://github.com/getsentry/vitest-evals --skill vitest-evals

Baixar ZIP GitHub

350

vitest-evals

Use the harness-backed API as the only authoring model.

First Steps

Read the package, app, or eval file being changed.
Identify the runtime target, then open only the needed reference.
Keep suites close to Vitest: one harness per describeEval(...), explicit run(...) inside each test, ordinary expect(...) assertions over the returned result.

Reference Router

Need	Open
Write or review a normal eval suite	`references/suite-authoring.md`
Build a custom app `Harness` without a first-party adapter	`references/custom-harness.md`
Integrate AI SDK `generateText`, `generateObject`, tools, or an AI SDK-style agent	`references/harness-ai-sdk.md`
Integrate a Pi AI or Pi Mono-style agent	`references/harness-pi-ai.md`
Add custom judges, suite judges, built-in judges, or `toSatisfyJudge(...)` assertions	`references/judges-and-assertions.md`
Assert on message, tool-call, or span history	`references/utilities.md`
Configure tool recording or replay	`references/tool-replay.md`
Diagnose failures, missing traces, odd output, or choose verification commands	`references/troubleshooting.md`

Runtime Defaults

Import describeEval(...), judges, and helpers from vitest-evals.
Bind exactly one harness to a suite.
Call run(input) where the test should execute the system.
Assert on result.output for app-facing behavior.
Use toolCalls(result) and message helpers for normalized session assertions.
Use spans(result), spansByKind(result, kind), and failedSpans(result) for span assertions.
Keep HarnessRun, NormalizedSession, usage, artifacts, and tool records JSON-serializable.
Report known tokens and USD cost in run.usage. Omit costUsd when unknown; use 0 only when the run is known to be free.
Keep rubric text and parsing in the judge. Put provider calls in a judgeHarness, then call them through ctx.runJudge(...) so usage is recorded.
Put scenario-owned criteria on the input value. Put direct-check expected values in Vitest case rows. Pass per-case judge criteria through explicit matcher options, and suite-wide criteria through judge config.
Custom judges should use createJudge(...) for stable reporter labels.

Verification

Prefer the smallest command that covers the edited files:

Task	Command
Lint file	`pnpm exec biome lint path/to/file.ts`
Format file	`pnpm exec biome format --write path/to/file.ts`
Test file	`pnpm exec vitest run path/to/file.test.ts -c vitest.config.ts`
Eval file	`pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts`
Type surface	`pnpm typecheck`
Package build	`pnpm build`