deepstream-run-mv3dt

Skill: Run DeepStream MV3DT

When to Use This Skill

Activate this skill when the user wants to set up, run, verify, or debug the DeepStream Multi-View 3D Tracking reference app. Typical prompts:

• "set up MV3DT DeepStream"
• "run the 4-camera MV3DT sample"
• "run the 12-camera MV3DT sample"
• "run MV3DT on my synchronized MP4s"
• "I have videos but no calibration; calibrate and run MV3DT"
• "show the BEV visualizer or Kafka metadata"
• "stop MV3DT" / "clean up MV3DT" / "tear down MV3DT"

Do not use this skill for single-view 3D tracking, generic DeepStream app development, or live-stream onboarding unless the user explicitly maps that work to this repo's MV3DT pipeline.

Examples

• "Deploy the DeepStream MV3DT 4-camera sample and show the OSD and BEV windows."
• "Run the MV3DT 12-camera sample headlessly with RTDETR and save videos."
• "Run MV3DT on synchronized MP4s under /data/mv3dt-demo using PeopleNetTransformer."
• "My custom MV3DT videos do not have calibration; use AutoMagicCalib, then run MV3DT."

Overview

Operate the Multi-View 3D Tracking reference app in DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view using the DeepStream Container path. The skill supports setup, shipped sample runs, custom synchronized MP4 datasets, calibration handoff to AutoMagicCalib, display/headless execution, OSD/BEV outputs, and Kafka metadata inspection.

Prerequisites

• MV3DT reference app directory on disk under DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view; if it is absent, ask before cloning the public DeepStream repo
• Docker with NVIDIA GPU support
• DeepStream container image access
• MV3DT sample datasets, models, custom parser libraries, Kafka, Mosquitto, and mv3dt_venv prepared by the repo setup script
• Working X11/VNC display for live OSD/BEV windows, or the saved-output headless path for tiled DeepStream MP4 plus Kafka-derived BEV MP4

Instructions

Step 0: Resolve MV3DT App Checkout

The skill can be installed outside the DeepStream repo. Resolve REPO_ROOT to the MV3DT app directory, not necessarily the Git top-level directory.

MV3DT_APP_SUBDIR="src/apps/reference_apps/deepstream-tracker-3d-multi-view"

is_mv3dt_app_dir() {
  test -f "$1/README.md" || return 1
  test -d "$1/config_templates" || return 1
  test -d "$1/scripts" || return 1
  grep -q "Multi-View 3D Tracking" "$1/README.md"
}

GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
CANDIDATES=()
if [ -n "${MV3DT_REPO_ROOT:-}" ]; then CANDIDATES+=("${MV3DT_REPO_ROOT}"); fi
if [ -n "${DEEPSTREAM_REPO_ROOT:-}" ]; then CANDIDATES+=("${DEEPSTREAM_REPO_ROOT}/${MV3DT_APP_SUBDIR}"); fi
CANDIDATES+=("${PWD}")
if [ -n "${GIT_TOP}" ]; then
  CANDIDATES+=("${GIT_TOP}" "${GIT_TOP}/${MV3DT_APP_SUBDIR}")
fi
CANDIDATES+=("${HOME}/DeepStream/${MV3DT_APP_SUBDIR}" "${HOME}/deepstream/${MV3DT_APP_SUBDIR}")

REPO_ROOT=""
for candidate in "${CANDIDATES[@]}"; do
  if [ -n "$candidate" ] && is_mv3dt_app_dir "$candidate"; then
    REPO_ROOT="$(cd "$candidate" && pwd)"
    break
  fi
done

if [ -z "${REPO_ROOT}" ]; then
  cat <<'EOF'
ERROR: MV3DT reference app directory was not found.
Set MV3DT_REPO_ROOT to an existing deepstream-tracker-3d-multi-view app directory, or ask the user to approve cloning the public DeepStream repo and then run:

  DEEPSTREAM_REPO_ROOT="${DEEPSTREAM_REPO_ROOT:-$HOME/DeepStream}"
  git clone https://github.com/NVIDIA/DeepStream.git "$DEEPSTREAM_REPO_ROOT"
  export MV3DT_REPO_ROOT="$DEEPSTREAM_REPO_ROOT/src/apps/reference_apps/deepstream-tracker-3d-multi-view"

Do not clone silently.
EOF
  exit 1
fi

cd "${REPO_ROOT}"
export REPO_ROOT MV3DT_REPO_ROOT="${REPO_ROOT}"

If the app directory cannot be resolved, ask the user for an existing checkout path or for approval to clone https://github.com/NVIDIA/DeepStream. Do not clone silently.

Step 1: Select The Primary Workflow

Load exactly one primary reference for the user's current request:

User intent	Reference
Install, prepare, or verify prerequisites	references/setup.md
Run bundled 4-camera or 12-camera sample	references/sample-run.md
Run custom synchronized MP4s	references/custom-dataset.md
Missing custom calibration	references/amc-calibration-handoff.md, then return to references/custom-dataset.md
View OSD, BEV, screenshots, recordings, or Kafka metadata	references/visualization-metadata.md
Stop a run, clean generated artifacts, or stop prerequisite services	references/setup.md

If setup, datasets, models, Kafka, Mosquitto, Docker GPU runtime, or the Python venv are missing, load references/setup.md before continuing to the user's original workflow.

When sample-run.md or custom-dataset.md needs to regenerate DeepStream configs, load references/generate-configs.md as the canonical shared config-generation reference rather than duplicating the shell logic.

Step 2: Follow The Run Stages

For every run, use this stage order:

Stage	Action
Validate	Check app directory, prerequisites, dataset shape, display/headless mode, Docker GPU support, and output-directory writability.
Prepare	State the selected sample/custom dataset, detector, run mode, output directory, expected output surfaces, and the Docker security decision for --privileged --net=host before launch.
Execute	Generate configs, start offline BEV capture first in headless mode or when saved BEV MP4 is explicitly requested, then run DeepStream or delegate missing calibration to AutoMagicCalib.
Verify	Confirm functional readiness with App run successful, fresh MP4 artifacts when file output is enabled, Kafka offsets/messages, and BEV message/frame counts when BEV MP4 capture runs.
Report	Summarize selected options, generated files, artifact paths, file sizes/counts, and any skipped or failed output surface; after a default 4-camera sample run, mention that the 12-camera sample is also available as a follow-up.

Step 3: Apply Defaults Explicitly

• Runtime path: DeepStream Container.
• Display path: if a working X11/VNC display is available, use the repo quick-start path with OSD and BEV windows. The OSD window does not save MP4 by default; when the user explicitly asks to save output in display mode, regenerate configs with both --enable-osd and --enable-file-output plus --enable-msg-broker.
• Headless path: if no working display is available, use saved outputs by default. Generate configs with --enable-file-output and --enable-msg-broker; start offline BEV capture before DeepStream so the run produces both the tiled DeepStream MP4 and the Kafka-derived BEV MP4.
• Sample dataset: support both shipped 4-camera and 12-camera datasets. If the user does not specify a sample, run the 4-camera sample first, then mention in the final report that the 12-camera sample is also available and can be run next.
• Detector: PeopleNetTransformer unless the user asks for RTDETR or PeopleNet2.6.3; carry the selected DETECTOR_MODEL through config generation and AMC camInfo modelInfo normalization. For custom calibration handoff, ask the user to choose the AutoMagicCalib detector instead of silently defaulting.
• Custom data source: synchronized MP4 files. Live-stream handling is outside this first-release skill.
• Calibration handoff: use standalone AutoMagicCalib skills instead of duplicating their setup or API workflow.

Step 4: Preserve Idempotency And User Data

• Readiness checks are safe to rerun.
• Setup may install packages, pull containers, download models, and start services; ask first.
• Config generation and DeepStream runs update generated files under EXPERIMENT_DIR; record RUN_STARTED_AT and do not report old artifacts as current-run success.
• Normal teardown stops only current run processes and leaves Kafka, Mosquitto, models, datasets, and generated artifacts in place unless the user explicitly asks to stop services or delete files.
• Custom datasets are user data. Copy missing calibration-format variants by default, and ask before renaming, overwriting, clearing, or deleting dataset files.

Success Criteria

• Prerequisite checks pass or the missing prerequisite is reported with a narrow next step.
• DeepStream run eventually prints App run successful.
• Display mode shows the DeepStream OSD grid and live BEV visualizer by default.
• Headless mode produces a fresh ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4 and attempts BEV MP4 capture by default.
• Kafka topic mv3dt receives current-run protobuf metadata when message broker output is enabled.
• BEV MP4 is reported as successful only when the separate BEV capture process produced nonzero messages and frames.

Key Output

• Generated configs: ${EXPERIMENT_DIR}/config_deepstream.txt, ${EXPERIMENT_DIR}/config_tracker.yml, ${EXPERIMENT_DIR}/config_msgconv.txt
• Saved DeepStream tiled MP4 when file output is enabled: ${EXPERIMENT_DIR}/outVideos/tiled_display_raw.mp4
• Saved BEV MP4 when offline capture is used: ${EXPERIMENT_DIR}/bev_outputs/trajectory_video_<timestamp>.mp4
• Kafka topic: mv3dt
• Sample output roots: ${REPO_ROOT}/experiments/deepstream/4cam and ${REPO_ROOT}/experiments/deepstream/12cam

Troubleshooting

Issue	First action
Setup prerequisites missing	Load references/setup.md and run the check-only path before setup.
Docker cannot access GPU	Fix NVIDIA Container Toolkit or Docker runtime before launching samples.
Display window missing	Check DISPLAY and /tmp/.X11-unix; use the headless saved-output path when no display is available.
DeepStream MP4 missing	Confirm configs were generated with --enable-file-output and verify the artifact is newer than RUN_STARTED_AT.
BEV MP4 missing or zero messages	Start offline BEV capture with --from-end before DeepStream, use a long enough --first-message-timeout, and verify Kafka offsets move during the run.
Kafka client shows no messages	Regenerate configs with --enable-msg-broker and verify topic mv3dt exists.
Custom dataset lacks calibration	Load references/amc-calibration-handoff.md; ask detector/settings choices before delegating.
Generated files are root-owned	Report the ownership issue and ask before applying a narrow generated-directory permission fix.

Safety Notes

• Ask before commands that use sudo, install packages, pull containers, download models, start or stop services, change host display access, overwrite dataset files, or clear generated state.
• Treat deleting Kafka, Mosquitto, models, datasets, or experiment outputs as destructive cleanup. Show the exact targets and get explicit confirmation before removing anything.
• Before running docker run --privileged --net=host, explicitly state that the container gets broad host, device, network, and mounted-repo access, then get user approval.
• Do not silently clone repositories, change host permissions outside the repo, rename or delete user datasets, or report old artifacts as current-run success.
• Treat custom videos, calibration, saved visualizations, and tracking metadata as potentially sensitive local data. Keep outputs local unless the user explicitly asks to move or share them.
• If permission fixes are needed for generated outputs, propose the narrowest generated-directory-only fix and ask first; never recommend broad world-writable recursive permission changes.

Related Skills

• amc-setup-calibration-stack - Launch the standalone AutoMagicCalib stack when calibration is needed.
• amc-run-video-calibration - Generate calibration from synchronized local MP4s before returning to MV3DT.