launching-evals

NeMo Evaluator Skill

Quick Reference

nemo-evaluator-launcher CLI

# Run evaluation
uv run nemo-evaluator-launcher run --config <path.yaml>
uv run nemo-evaluator-launcher run --config <path.yaml> -t <a_single_task_to_be_run_by_name>
uv run nemo-evaluator-launcher run --config <path.yaml> -t <task_name_1> -t <task_name_2> ...
uv run nemo-evaluator-launcher run --config <path.yaml> -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 ...

# Preview the resolved config and the sbatch script without running the evaluation
uv run nemo-evaluator-launcher run --config <path.yaml> --dry-run

# Check status (--json for machine-readable output)
uv run nemo-evaluator-launcher status <invocation_id> --json

# Get evaluation run info (output paths, slurm job IDs, cluster hostname, etc.)
uv run nemo-evaluator-launcher info <invocation_id>

# Copy just the logs (quick — good for debugging)
uv run nemo-evaluator-launcher info <invocation_id> --copy-logs ./evaluation-results/

# For artifacts: use `nel info` to discover paths. If remote, SSH to explore and rsync what you need.
# If local, just read directly from the paths shown by `nel info`.
# ssh <user>@<hostname> "ls <artifacts_path>/"
# rsync -avzP <user>@<hostname>:<artifacts_path>/{results.yml,eval_factory_metrics.json,config.yml} ./evaluation-results/<invocation_id>.<job_index>/artifacts/

# Resume a failed/interrupted run (re-sbatches existing run.sub in the original run directory)
uv run nemo-evaluator-launcher resume <invocation_id>

# List past runs
uv run nemo-evaluator-launcher ls runs --since 1d   

# List available evaluation tasks (by default, only shows tasks from the latest released containers)
uv run nemo-evaluator-launcher ls tasks
uv run nemo-evaluator-launcher ls tasks --from_container nvcr.io/nvidia/eval-factory/simple-evals:26.03

Workflow

The complete evaluation workflow is divided into the following steps you should follow IN ORDER.

1. Create or modify a config using the nel-assistant skill. If the user provides a past run, use its config.yml artifact as a starting point.
2. Run the evaluation. See references/run-evaluation.md when executing this step.
3. Monitor progress (MANDATORY after every nel run): poll status repeatedly until SUCCESS/FAILED. See references/check-progress.md.
4. Post-run actions (when terminal state reached):
   1. When the evaluation status is SUCCESS, analyze the results. See references/analyze-results.md when executing this step.
   2. When the evaluation status is FAILED, debug the failed run. See references/debug-failed-runs.md when executing this step.

Key Facts

• Benchmark-specific info learned during launching/analyzing evals should be added to references/benchmarks/
• PPP = Slurm account / project portfolio code (the account field in cluster_config.yaml). When the user says "change PPP to X", update the account value (e.g., <old_account> → <new_account>).
• Slurm job pairs: NEL (nemo-evaluator-launcher) submits paired Slurm jobs — a RUNNING job + a PENDING restart job (for when the 4h walltime expires). Never cancel the pending restart jobs — they are expected and necessary.
• HF cache requirement: For configs with HF_HUB_OFFLINE=1, models must be pre-downloaded to the HF cache on each cluster before launching. Before running a model on a new cluster, always ask the user if the model is already cached there. If not, on the cluster login node: python3 -m venv hf_cli && source hf_cli/bin/activate && pip install huggingface_hub then HF_HOME=<your_hf_cache_path> hf download <model> (on lustre-style HPC clusters this is typically under /lustre/.../<group>/users/<username>/cache/huggingface). Without this, vLLM will fail with LocalEntryNotFoundError.
• data_parallel_size is per node: dp_size=1 with num_nodes=8 means 8 model instances total (one per node), load-balanced by haproxy. Do NOT interpret dp_size as the global replica count.
• payload_modifier interceptor: The params_to_remove list (e.g. [max_tokens, max_completion_tokens]) strips those fields from the outgoing payload, intentionally lifting output length limits so reasoning models can think as long as they need.
• Auto-export git workaround: The export container (python:3.12-slim) lacks git. When installing the launcher from a git URL, set auto_export.launcher_install_cmd to install git first (e.g., apt-get update -qq && apt-get install -qq -y git && pip install "nemo-evaluator-launcher[all] @ git+...#subdirectory=packages/nemo-evaluator-launcher").
• Do NOT use nemo-evaluator-launcher export --dest local — it only writes a summary JSON (processed_results.json), it does NOT copy actual logs or artifacts despite accepting --copy_logs and --copy-artifacts flags. nel info --copy-artifacts works but copies everything (very slow for large benchmarks). Preferred approach: use nel info to discover paths — if local, read directly; if remote, SSH to explore and rsync only what you need. Note that nel info prints standard artifacts but benchmarks produce additional artifacts in subdirs — explore to find them.