cultivar

cultivar

cultivar is a CLI that measures whether an agent skill actually improves an agent's behavior. For each task it runs the agent with the skill and without it (and optionally with the source docs), then an LLM grader scores each run against a natural-language rubric. Use this skill when the user wants to create, run, or interpret cultivar evals.

The loop

1. cultivar init <skill> — scaffold tasks/<skill>.yaml + a SKILL.md stub.
2. Edit the task file (intent + PASS/FAIL criteria) and the skill.
3. cultivar run -s <skill> -r <runner> --grade — run all variants and grade.
4. cultivar report / cultivar show latest -t <task> — read the outcome.
5. Iterate on the skill; re-run; compare.

Always confirm the install first with cultivar hello (or cultivar hello --no-grade when no ANTHROPIC_API_KEY is available) — it runs a packaged smoke task end-to-end.

Commands

• cultivar init <skill> [--skills-dir DIR] — scaffold task YAML + SKILL.md stub.
• cultivar run -s <skill> -r <claude|copilot|gemini> — run. Key flags:
  • -t <task> one task · -v <variant> one variant (see Variants below for the full list)
  • --remote run in isolated Modal sandboxes · -n N repeat · -p N parallelism
  • --grade grade after running · --title NAME label the run · --dry-run print the prompt + command without calling anything · --timeout S per-call budget (default 90)
  • --model ID pin the agent's model (Claude runner only, e.g. claude-sonnet-5); unset uses the CLI's own default. A task's extra_tools: [WebSearch, WebFetch] field unions extra tools onto whichever variant it requests.
• cultivar grade <run|latest> -s <skill> [--report] — (re)grade an existing run. --model picks the grading model (any current Claude model works, including the "-5" generation) · --max-tokens raises the per-reply budget if evidence/reasoning truncate.
• cultivar report [run] — summary table across runners/variants.
• cultivar show <run|latest> -t <task> [--grader|--conversation-only|--workdir] — inspect one run.

--dry-run is the safe way to preview exactly what will be sent before spending tokens.

Variants (the controls)

• with-skill — skill loaded; prompt prefixed Use the /<skill>.
• without-skill — no skill; identical otherwise. The baseline.
• with-docs — no skill, but the task's ground_truth.context_refs files are prepended. Only runs for tasks that declare context_refs. Entries can be local files or http(s):// URLs (fetched live, cached under ./.docs_cache).

Read two deltas: with-skill vs without-skill ("does the skill do anything?") and with-skill vs with-docs ("is the distilled skill better than dumping the raw docs?").

Docs-eval extras (Claude runner only)

Three more real --variant choices, opt-in only, not part of the default "run every variant" sweep:

• without-docs — identical to without-skill, named for docs-testing clarity (this is what "no docs at all" actually means when the thing under test is a doc, not a skill).
• self-navigate — no injected reference material, but WebFetch is enabled and ground_truth.self_navigate_refs gives it a starting page. Tests whether the agent can find the right doc on its own; with-docs tests whether the content is good once handed over. Different questions, keep them separate.
• with-docs:<label> — set ground_truth.doc_versions: {label: [refs]} instead of flat context_refs; --variant with-docs then auto-expands into one real run per version, graded and reported side by side. Compares two doc versions (e.g. before/after a rewrite) in one command instead of two separate runs you diff by hand.

Tasks

tasks/<skill>.yaml holds one or more tasks. Each task:

tasks:
  - id: a-short-id
    intent: "what you'd ask the agent to do"
    category: cli            # or: code-gen
    # setup / teardown / verify: optional shell hooks
    # env: ["SOME_KEY"]      # required env vars, checked upfront
    # extra_tools: [WebSearch, WebFetch]  # unioned onto whichever variant is requested
    ground_truth:
      criteria: |
        PASS requires <2-3 concrete, checkable things>.
        FAIL if <a common failure mode>.
      # context_refs: [docs/ref.md]      # activates the with-docs variant
      # self_navigate_refs: [docs/overview.md]  # activates self-navigate (a starting point, not the answer)
      # doc_versions: {v1: [old.md], v2: [new.md]}  # activates with-docs:v1 / with-docs:v2 instead of flat with-docs

Guidance:

• For code-gen tasks, the intent must say "write a file … in the current directory." Anything the agent writes to its cwd is captured and shown to the grader. A code-gen task that produces no file auto-fails.
• Write criteria as crisp PASS conditions + at least one concrete FAIL mode — vague criteria produce vague grades.

Where skills live

cultivar tests exactly one skill per run (the -s one). It resolves the skills root as: --skills-dir flag → CULTIVAR_SKILLS_DIR env → ./.claude/skills. Keep skills-under-test outside .claude/ (e.g. ./skills, via CULTIVAR_SKILLS_DIR=skills) if you don't want your interactive coding agent to auto-load them.

Local vs remote

• Local (default) — uses the runner CLI installed on your machine + its auth.
• --remote — each (task, variant, repeat) runs in its own Modal sandbox: clean isolation, parallelism, reproducibility. Requires a Modal account (modal token new) and a secret holding the agent's ANTHROPIC_API_KEY (default secret name eval-sandbox-secrets; override with CULTIVAR_MODAL_SECRET). Prefer --remote for rigorous comparisons. The grader always runs locally and needs ANTHROPIC_API_KEY.

Reading results

results/<timestamp>[__title]/ holds per-run .json (stats), .md (readable trace), .jsonl (raw events), and .workdir/ (files the agent wrote). grades.json holds the verdicts. Use cultivar report for the table and cultivar show … --grader for the grader's reasoning + suggestions on a failure.

Gotchas

• Grading needs ANTHROPIC_API_KEY (loaded from a .env in the cwd). hello --no-grade and run --dry-run need no key.
• A grader call that fails is recorded as a FAIL for that conversation and the run continues. Auth and permission errors abort the whole grading run immediately.
• tasks/, examples/, and results/ are cwd-relative and user-owned.
• One run is a sample. Use -n 3 (or more) for anything you'll act on.