evaluation

NeMo Evaluator Launcher Assistant

Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress. NEL manages serving for canaries and full runs; see Step 3's serving-handoff check for external endpoints.

Completed-run analysis

For completed runs, skip launch/config; use launching-evals for artifacts/analysis and apply run-validation.md before scores. Its policy overrides blanket uncapping and fixed token/context limits.

Workspace integration

If MODELOPT_WORKSPACE_ROOT is set, use the common skill's workspace-management.md and reuse existing workspaces (PTQ → checkpoint smoke test → NEL evaluation; carry compatibility requirements into NEL's deployment config).

Workflow

- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)
- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)
- [ ] Step 2: Build base config (5-question flow OR shortcut)
- [ ] Step 3: Configure deployment (model path, params, cross-check)
- [ ] Step 4: Fill remaining ??? values
- [ ] Step 5: Confirm tasks (iterative)
- [ ] Step 6: Multi-node (if needed)
- [ ] Step 7: Interceptors (if needed)
- [ ] Step 7.5: Container auth (SLURM private images)
- [ ] Step 8: Dry-run → canary → full run
- [ ] Step 9: Verify completed run

nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST

A few agentic AA benchmarks do not run on the currently validated nemo-evaluator-launcher 0.2.6 path (Steps 1–9 don't apply). They run on nel-next (nemo-evaluator[harbor] 0.4.x) — a separate package, CLI (nel eval run), -O overrides, and services/benchmarks/cluster/output schema. If the user asks for one, do not add it to a 0.2.6 evaluation.tasks list — instead:

1. Read references/nel-next.md (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md; start from recipes/examples/example_eval_next.yaml.
2. Isolated nel-next venv: "$SKILL_DIR/scripts/nel-next.sh" --setup-only (keeps 0.2.6 nel untouched).
3. Run modelopttools:eval-config (Step 3b) to write the AWS-sandbox creds + harbor infra rows (${NEL_NEXT_EVAL_IMAGE}, ${HARBOR_*_ECR_REPOSITORY}, ${HARBOR_ECS_REGION}) into .env; always include the output.export_config.mlflow block.
4. Dry-run → canary → full (nel-next.sh eval run), then push to MLflow — SLURM doesn't auto-export, so run nel-next.sh mlflow-push -r <run_id> -c <cfg> after (config-driven; see references/nel-next.md).
5. Before reporting a full-run score, apply references/run-validation.md, including its benchmark-specific checks.

Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.

MRCR (NeMo Gym simple_agent) path — branch here too

MRCR does run on the currently validated 0.2.6 nel launcher (as a nemo_gym task, not nel-next), so Steps 1–9 apply — but it is mechanically special and standalone (one gym eval per config; never mix it with aa/ tasks). It is simple as gym tasks go: simple_agent, no judge — deterministic prefix-gated grading, HF_TOKEN the only secret. Not an AA benchmark — never generate it for an "AA" request. If the user asks for MRCR:

1. Read references/gym.md (pinned launcher, gym prepare/reap machinery, pin↔container coupling, preflight gaps, failure modes) + recipes/tasks/gym/mrcr.md; start from recipes/examples/gym/example_mrcr.yaml (1M variant, like the golden).
2. Pick the variant first (config_n3_1m / config_n3_128k / config) — it sets the context cap, dataset and metric prefix; the three are not comparable; set it in both data_prep_params and collect_rollout_params. Take the largest variant within the checkpoint's trained context — see the recipe.
3. .env: HF_TOKEN (dataset + n3 tokenizer are gated) plus NEMO_EVALUATOR_TRUST_PRE_CMD=1 (the pre_cmd installs tiktoken + transformers; prepare fails without it) and NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1 (nemo_gym is not in the FDF map).
4. MRCR needs a git-backed Gym image. The pin is newer than any image's baked Gym and must apply, so the template's container: is ??? and the bootstrap exits 1 on a non-git /opt/Gym (the public eval-factory/nemo-gym:* images). NVIDIA-internal: modelopttools:eval-config Step 3d names a working image.
5. Long-context deploy: for 1M, --max-model-len 1100000 + VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; for 128K, --max-model-len at the checkpoint's trained context (room for the answer above 131,072) and no override. 1M also needs gpu_memory_utilization: 0.95. Both: multi-instance fan-out; never cap output tokens; report the needle-count strata alongside pass@1/accuracy.
6. Run both dry-run and launch through "$SKILL_DIR/scripts/nel-gym.sh"; it enforces the currently validated 0.2.6 launcher even if nel on PATH is stale and avoids an unset NEL_INVOCATION_ID failure before client startup. Canary with limit_samples — the MRCR template forwards it as ++limit (see the recipe's Canary section); the prepare pass still runs in full.

Detailed launcher instructions live in launcher-workflow.md. Read only the sections needed for the current stage; retain the dry-run → canary → full-run gates. Existing configs must pass Step 3's serving-handoff check before Step 8. Paths in that reference are relative to this skill directory.

Step 1 — Prerequisites

Read Step 1.

Step 2 — Build base config (when not using shortcut)

Read Step 2.

Step 3 — Configure deployment

Read Step 3.

Step 4 — Fill remaining ??? values

Read Step 4.

Step 5 — Confirm tasks (iterative)

Read Step 5.

Step 6 — Multi-node

Read Step 6.

Step 7 — Interceptors

Read Step 7.

Step 7.5 — Container registry auth (SLURM private images only)

Read Step 7.5.

Step 8 — Run evaluation (gated dry-run → canary → full)

Read Step 8.

Step 9 — Verify completed run

Apply run-validation.md's Bounded Evaluation-Failure Policy and Timeout and Output-Limit Accounting before reporting scores. For comparisons, also apply its External Baseline Sanity Check, then use compare-results.

Issues: https://github.com/NVIDIA-NeMo/Evaluator/issues · https://github.com/NVIDIA-NeMo/Evaluator/discussions