benchmark-isaacsim

Isaac Sim Benchmarking

Parameter references may be outdated. Always verify with ./python.sh <script> --help. For profiling details (Tracy, Nsight), see the profiling skill.

Setup

See the install-isaacsim skill for installation (pip, source build, Docker).

Before Running Any Benchmark

1. Use a WARM run for headline FPS/frametime — see the COLD/WARM/TRACY method in the profiling skill.
2. Set CPU governor to performance — see perf-tuning skill.
3. Set Nucleus auth if using Nucleus-hosted assets — see install-isaacsim skill.
4. Do not patch Isaac Sim shutdown by default. If Tracy shutdown hangs after outputs are complete, use the scoped last-resort guidance in the profiling skill.

Benchmark Scripts

All in standalone_examples/benchmarks/. Run via ./python.sh <script> [args].

Common params: --num-frames (default 600), --num-gpus, --backend-type, --viewport-updates, --non-headless

Output control Kit args (append to any benchmark command):

--/exts/isaacsim.benchmark.services/metrics/metrics_output_folder=<output_dir>
--/log/file=<output_dir>/kit.log

Critical Gotchas

Parameter format: HYPHENS, not underscores

Isaac Sim uses hyphens in parameter names. Underscores are silently ignored by argparse:

CORRECT: --num-cameras 8 --num-gpus 1 --num-frames 600
WRONG:   --num_cameras 8  (silently uses default=1!)

Always verify result JSON to confirm params were applied.

Headless and viewport validation

When comparing benchmark outputs, record whether the run used --non-headless, --viewport-updates, render products, or camera sensors. Do not change viewport code or settings from this skill; if extra viewport work appears to affect results, hand off to perf-tuning.

JWT token expiry = silent black renders

Expired OMNI_PASS tokens cause silent asset loading failure. See install-isaacsim skill for the expiry check command.

carb.log_warn() doesn't go to stdout

Monitor progress by polling for result JSON files, not watching stdout. Kit log: ~/.nvidia-omniverse/logs/Kit/isaacsim*/kit.log or --/log/file=/tmp/kit.log.

Hung processes after results are written

If kpis_*.json and *.tracy exist with non-zero size and the process has not exited after 2 minutes with no new output, follow the profiling skill's guide-aligned shutdown handling and force-kill the app and capture processes.

Optimization Handoff

If results indicate per-camera viewport overhead, too many render products, default viewport work in headless mode, or poor multi-GPU scaling, do not apply fixes here. Summarize the benchmark evidence and use perf-tuning for the configuration changes.

Output Files

• kpis_benchmark_<name>.json — benchmark KPIs (main results)
• kit.log — execution log (if --/log/file= is set)
• *.tracy — Tracy profiling trace (only with Tracy args)
• *.nsys-rep — Nsight profiling trace (only with nsys)

Note: benchmark_result.json and startup_*.json are NOT produced by default in v5.x/v6.x. The primary results file is kpis_benchmark_<name>.json.