nre

NRE — NVIDIA Omniverse NuRec (Neural Reconstruction Engine)

Purpose

Drive the public NVIDIA Omniverse NuRec / Neural Reconstruction Engine containers (nvcr.io/nvidia/nre/nre, nvcr.io/nvidia/nre/nre-tools) to train a 3DGUT/3DGRT Gaussian reconstruction from an NCore V4 camera+LiDAR clip, render novel views (locally or via gRPC), generate aux data, export PLY/depth/mesh/ego-mask/tracks, package Asset Harvester output into a USDZ, and evaluate rendering metrics.

This skill carries the host-side toolkit around the NRE CLI: NGC credential resolution, cached-image notes, local render recipes, MP4 encoding, warm serve-grpc boot/teardown scripts, a thin Python gRPC client for repeated RGB renders, bundled rig JSONs, pre-baked custom-rig trajectories, and bash / Hydra / OSMO workflow templates.

When to Use / When NOT to Use

Use this skill when the user has an NCore V4 clip (or a USDZ + NRE artifact pair) on a Linux x86_64 host with an NVIDIA GPU and an NGC API key, and wants to train, render, generate aux data, export artifacts, insert/remove actors, run the gRPC server, or evaluate metrics. Concrete triggers:

• Train a multi-camera + LiDAR AV clip into a renderable USDZ scene with 3DGUT (or 3DGRT ray-traced) Gaussians.
• Generate NuRec auxiliary data (seg, depth, ego mask, DINOv2, LiDAR-seg visibility) using nre-tools.
• Render frames locally (no server) along the training rig or a custom rig + offsets.
• Render novel views via the sensorsim gRPC API (CARLA, Isaac Sim, AlpaSim, custom simulator), optionally with Difix artifact-removal.
• Render LiDAR sweeps via render-grpc --lidar.
• Export PLY / ego masks / depth / Poisson mesh / ground mesh / point clouds / cuboid tracks / NCore tracks / custom rig trajectories.
• Insert / remove / replace 3D actors with export-external-assets
  • render-grpc --edit-assets.
• Render the gated HF dataset nvidia/PhysicalAI-Autonomous-Vehicles-NuRec.
• Upgrade an old USDZ once (upgrade-artifact).
• Inspect / evaluate (export-parsed-config, gaussian-statistics, eval-rendering-metrics, compute-metrics, eval-ground-mesh).
• Browse a USDZ or PLY in the in-container viewer.

Do NOT use this skill when:

• The user still needs to convert raw sensor data into NCore V4 (use the ncore skill first; NRE consumes NCore-formatted shards).
• The user wants per-object 3D asset extraction from sparse views (use asset-harvester; NRE only consumes AH outputs via export-external-assets).
• The user only needs to clean up already-rendered frames using the standalone Cosmos-based Fixer (use nurec-fixer). NRE's inline --enable-difix flag is still on this skill's surface, but the standalone harmonizer pipeline is owned by nurec-fixer.

Inputs

• dataset_dir — host directory holding the NCore shards (<NAME>.zarr.itar, <NAME>.json, and any pre-generated <NAME>.aux.*.zarr auxiliary shards). Required.
• dataset_name — basename of the NCore dataset (the part before .zarr.itar). Required.
• output_dir — host directory NRE will fill with checkpoints, parsed config, metrics, videos, and USDZ artifacts. Required.
• camera_ids / lidar_ids — sensor IDs from the NCore JSON to include. Default: all sensors per recipe.
• config_name — Hydra config path resolved inside the container. Pick by source dataset:
  • Waymo Open Dataset → configs/apps/AV/Waymo/3dgut_dynamic.yaml (and its _mcmc / _road_semantic / _static siblings). These are Waymo-only — they bake in the Waymo sensor rig and conventions.
  • NVIDIA Physical AI Autonomous Vehicles (PAI) → /apps/prod/Hyperion-8.1/car2sim_6cam.yaml (the Hyperion-8.1 car2sim 6-camera recipe used by the Maglev PAI pipeline). Typically referenced via the small overlay shipped at references/configs/pai.yaml, which extends car2sim_6cam.yaml with PAI's lidar_top_360fov ID, six-camera validation set, and lidar intensity supervision; mount it as {nre_config_dir}/external_overrides.yaml and pass --config-name=external_overrides.
  • PandaSet / NVIDIA AV (NV) / Tesla / Alpasim → see the matching configs/apps/AV/{PandaSet,NV,Tesla}/… or configs/apps/Alpasim/… recipes in references/configuration.md.
  • Do not use the Waymo recipes for PAI clips — the sensor rig, validation cameras, and lidar IDs differ.
• mode — train, val, or trainval. Default: trainval.
• NGC_API_KEY — required env var. Generate at https://org.ngc.nvidia.com/setup/api-keys.

Instructions

1. Validate prerequisites. Have the agent execute scripts/validate_setup.py via its standard script runner (run_script("scripts/validate_setup.py"), or python scripts/validate_setup.py [--strict]). It checks Docker, NVIDIA Container Toolkit, GPU/driver, and NGC_API_KEY. Resolve any FAIL line before pulling the image.
2. Authenticate Docker to NGC + pull the public containers. See references/install.md. Pull both nvcr.io/nvidia/nre/nre:latest and nvcr.io/nvidia/nre/nre-tools:latest.
3. Confirm input layout. The dataset directory must contain <NAME>.zarr.itar, <NAME>.json, and any <NAME>.aux.*.zarr shards. If the NCore data is fresh, generate auxiliary data first — see references/aux-data.md.
4. Train / validate the reconstruction. Run the train recipe in references/cookbook.md with the chosen Hydra --config-name, mode, dataset.path, and out_dir. For multi-GPU append trainer.world_size=<N> trainer.num_nodes=<M> (see Workflow D). Set checkpoint.artifact.enabled=true if you intend to render or serve the result.
5. Export downstream artifacts. Use export sub-commands (export-gaussian-plys, export-mesh, export-ground-mesh, export-ego-mask, export-depth, export-sequence-tracks, export-ncore-tracks, …) — full surface in references/cli-reference.md.
6. Render novel views — pick the backend.
   • Local CLI — nre render --artifact-path <usdz> writes frames on disk along the training trajectory, with optional rig offsets or --custom-rig-trajectory. No gRPC server. See references/local-render.md.
   • Warm RGB service — boot serve-grpc once with scripts/session_warm_server.sh, extract protobuf stubs, and use references/NRE_RenderClient/scripts/thin_client.py for repeated single-camera or batch_render_rgb calls.
   • Remote CLI / simulator integration — serve-grpc + render-grpc (or your own client via nre.grpc.protos). Required for LiDAR rendering, simulator loops, Difix, or --edit-assets. See references/grpc-api.md and references/physical-ai-render.md.
7. Edit actors (optional). Run export-external-assets to repackage Asset-Harvester output into a new USDZ, then pass the produced edit-assets.json to render-grpc --edit-assets (with serve-grpc --enable-editing-actors). See references/asset-editing.md.
8. Validate the result. Confirm <output_dir>/<RUN-ID>/usd-out/last.usdz opens, metrics.yaml reports a reasonable test/psnr, and the generated MP4s render. For more thorough metrics use Workflow I (eval) in references/workflows.md. Tear down any gRPC server (Ctrl-C or docker rm -f).

For any NRE task expected to run 5 minutes or longer (training, OSMO jobs, multi-clip renders), follow references/long-running-tasks.md: delegate to a subagent / background job and report compact status at least every 5 minutes.

Examples

Example 1 — End-to-end NCore → USDZ → render

Walk Workflow A in references/workflows.md: validate host → generate aux data → train (cookbook recipe) → export → local render or serve-grpc. Concrete commands live in the referenced files; this index does not duplicate them.

Example 2 — Skip training, render the gated Physical AI dataset

Walk Workflow B: download nvidia/PhysicalAI-Autonomous-Vehicles-NuRec from HuggingFace, then jump to serve-grpc + a Python client. Coordinate-frame conversion code is in references/physical-ai-render.md.

Example 3 — Insert Asset-Harvester actors into a USDZ

Walk Workflow C: run asset-harvester, then export-external-assets, edit edit-assets.json, and call serve-grpc --enable-editing-actors + render-grpc --edit-assets. Schema lives in references/asset-editing.md.

Example 4 — Warm-server thin-client for repeated RGB renders

Walk the warm-server quick start at the bottom of references/workflows.md. Boot scripts/session_warm_server.sh, render with thin_client.py, tear down with scripts/session_teardown.sh.

Backend Selection

Pick the smallest backend that exposes the requested feature:

• Local Docker, single command. Use nre render, render-grpc, or an export sub-command directly. Simplest for one-off renders, LiDAR sweeps, actor edits, rolling shutter, in-container video export, or exact --replicate-training-views behavior. See references/local-render.md, references/nre-image-notes.md, and references/mp4-encoding.md.
• Local Docker, warm serve-grpc + thin host client. Use for render-heavy RGB sessions where repeated Docker/Python/CUDA cold-start dominates latency, or where multiple cameras should be rendered through one batch_render_rgb RPC. See references/NRE_RenderClient/README.md and scripts/session_warm_server.sh / scripts/session_teardown.sh.
• OSMO / cluster workflows. Use the templates under references/example-workflows/osmo/ for multi-clip fan-out, isolation from the local machine, or training jobs that should not run on the user's workstation. Follow references/ngc-and-registry.md for registry credentials and references/long-running-tasks.md for polling discipline.

Output Format

Structured deliverables placed under ${output_dir}/${RUN_ID}/ by the NRE container (no JSON state file required from the agent):

• config/parsed.yaml — Hydra-resolved training config.
• checkpoints/last.ckpt (plus periodic snapshots).
• val/metrics.yaml — per-frame PSNR / SSIM / LPIPS under test/*.
• val/*.mp4, val/<frame>/*.png — depth, opacity, segmentation, RGB visualisations.
• usd-out/last.usdz — USDZ containing the trained reconstruction, data_info.json, rig_trajectories.json, sequence_tracks.json, parsed_config.yaml, checkpoint.ckpt, optional mesh.ply, and map.xodr. Render with nre render, serve-grpc + render-grpc, the in-container viewer, or hand to a downstream simulator (CARLA, AlpaSim, Isaac Sim).
• *.ply / ego_mask/* / depth/* / sequence_tracks.json / ncore_tracks.json / mesh.ply / ground_mesh.ply — produced by the matching export sub-command.

Scripts

References

Read these on demand; keep SKILL.md as the routing layer.

• references/install.md — docker login nvcr.io, image pull, full prerequisite matrix, and safe secret-handling for NGC_API_KEY / HF_TOKEN.
• references/cookbook.md — most-used docker run invocations: train + validate, re-validate with shift, local render at quarter or native res, serve-grpc boot, LiDAR sweep, in-container --help.
• references/workflows.md — workflows A – I end-to-end, plus the warm-server thin-client quick start.
• references/troubleshooting.md — extended error matrix (OOM, wandb blocking, --artifact-glob mismatches, deprecated flags, gRPC LiDAR size, etc.).
• references/teardown.md — disk cleanup, post-teardown verification, ownership-recovery.
• references/cli-reference.md — full sub-command surface of the NRE container (training, validation, render, serve-grpc, render-grpc, render-novel-trajectory, every export-*, upgrade-config / upgrade-artifact, gaussian-statistics, eval-rendering-metrics, compute-metrics, viewer, ply_viewer, profile-dataloader, run-script, the nre-tools aux-data + AH entry points).
• references/configuration.md — Hydra recipe map for Waymo / NV / PandaSet / Tesla / Alpasim, plus override matrix.
• references/aux-data.md — nre-tools auxiliary-data CLI.
• references/local-render.md — host-side docker run … render recipes for rig offsets and export-custom-rig-trajectory.
• references/NRE_RenderClient/README.md — warm-server thin Python gRPC client.
• references/grpc-api.md — sensorsim gRPC server flags + Python client cookbook.
• references/nre-image-notes.md — cached-image discovery, 26.04+ vs 26.03 vs pre-26.03 flags.
• references/ngc-and-registry.md — NGC API key resolution.
• references/mp4-encoding.md — host-side ffmpeg recipe.
• references/asset-editing.md — export-external-assets + edit-assets.json schema.
• references/physical-ai-render.md — recipe for rendering the HuggingFace NuRec dataset.
• references/example-workflows/ — bash, Hydra, and OSMO templates.
• references/rig-json/ — bundled rig.json and augmented_rig.json.
• references/custom-rig-trajectories/ — pre-baked export-custom-rig-trajectory outputs.
• references/long-running-tasks.md — background-job + 5-minute status reporting convention.
• references/nurec-skill-catalog.md — routing table for sibling NuRec-stack skills.
• Public product page: https://www.nvidia.com/en-us/omniverse/nurec/
• HF dataset: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec
• HF Fixer model: https://huggingface.co/nvidia/Difix3D
• NGC Fixer model card: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/models/nurec-fixer

Prerequisites

Linux x86_64 + NVIDIA GPU + Docker 23+ + NVIDIA Container Toolkit 1.13+ + NGC_API_KEY. Full matrix (driver minimums per arch, shm-size, file ownership, GPU-tier guidance) lives in references/install.md. Always verify via scripts/validate_setup.py before pulling the image.

Limitations

• Linux x86_64 only. aarch64 (e.g. Jetson) is not supported.
• Internal source not redistributable. Use only the public NGC containers and the public NuRec docs.
• Multi-GPU defaults are conservative. Set trainer.world_size / trainer.num_nodes explicitly to scale out; SLURM is auto-detected when both are 0. Quality plateaus past ~6 GPUs; per release notes, multi-GPU + dataset.aux_data=false is a known crash combination.
• --config-name paths differ between train and val/export. Training uses container-bundled recipes; validation and exports re-pass the parsed.yaml written under <output_dir>/<RUN-ID>/config/.
• Validation may prompt for wandb. Choose option 3 to skip in non-interactive runs, or pass logger=tensorboard / logger=dummy.
• Render gRPC is data-format-pinned. Older releases warn / reject artifacts that pre-date them; check release notes when mixing client / server versions.
• Asset-Harvester input only. export-external-assets requires AH outputs; raw .ply files won't carry the per-asset cuboid metadata.
• Difix variants are pluggable. The container ships both the Cosmos Difix variant (default since 25.09 — difix=cosmos_difix) and the legacy Stable-Diffusion variant (difix=sd_difix). The newer Cosmos-Predict-based Fixer variants live in the nurec-fixer skill.
• render ↔ render-grpc overlap. render runs in-container without a server; render-grpc requires an active serve-grpc. Use render for batch novel-view jobs and render-grpc when you need actor editing, LiDAR rendering, or a long-lived service.

Troubleshooting (top 4)

Full matrix in references/troubleshooting.md.

Teardown

Full inventory, ownership-recovery, and post-teardown verification commands live in references/teardown.md. Headline: stop serve-grpc containers, docker image rm nre/nre-tools, rm -rf ${HOME}/.cache/nre and your <output_dir>/<RUN-ID>/. Do not revoke NGC_API_KEY unless you suspect it has been leaked.