rtvi-cv-scaffold-vss-service

Construct RTVI VSS CV Service

Scaffolds a deployable custom perception microservice that:

1. runs a DeepStream pipeline with YOLO26 primary inference + tracker,
2. converts detection metadata to the VSS protobuf schema and publishes it to the mdx-raw Kafka topic, and
3. drops into the existing VSS compose stack via the bp_developer_search_2d and bp_developer_alerts_2d_cv profile flags so the downstream Search Workflow, Alert Verification, and Behavior Analytics services consume it without further changes.

The scaffolded output is a runnable repo, not a design document.

Scope boundaries

This skill name describes the service type (a VSS-bound RTVI CV microservice), not a model family. What is generic vs YOLO26-specific:

Do not treat the scaffold as a model-agnostic generator. Agents should either run the YOLO26 scaffold as-is or consciously edit pgie/compose paths for another detector while keeping the VSS contract fixed.

When to use

Use this skill when the user wants to:

• replace VSS's default perception service (RT-DETR / GDINO / YOLOv11) with a custom YOLO26-based detector,
• add new object classes or domain-specific tracking logic while keeping the VSS Search / Alerts / Behavior Analytics workflows intact,
• ship a perception microservice on a customer's hardware that integrates with a VSS deployment they manage.

Do not use this skill to:

• deploy or operate VSS itself (use VSS deployment runbooks),
• swap the detector in the default vss-rt-cv container without a new microservice (use rtvi-cv-customize-model).

This skill assumes a target VSS deployment already exists or will be brought up separately.

Instructions

• Read references/integration-contract.md first and keep mdx-raw, msg-conv-payload-type=2, broker reachability, and the protobuf-2 contract fixed unless the user explicitly wants to break VSS compatibility.
• Treat all VSS deployment paths as relative to a separate checkout of the public VSS Blueprint repository, not the DeepStream repository. Clone or reuse a VSS checkout that is v3.2.1 or compatible, then deploy that release (see VSS Quickstart).
• If the user does not have the YOLO26 ONNX, labels file, or custom parser .so, say so plainly: scaffolding and unit tests can proceed with placeholders, but the live DeepStream app cannot run yet. Point them at Ultralytics/Hugging Face for pretrained weights and ONNX export (see references/yolo26-deepstream.md).
• When assets are missing, stop the "live validation" path at scaffold generation plus host-only unit tests. Do not imply that docker compose up, deepstream-app, or kafka_smoketest.py can succeed without the customer-supplied ONNX, labels matching num-detected-classes, and a parser exposing NvDsInferParseYolo26.
• Start the generated service only after the VSS Kafka topic initializer has completed. Keep it as a separate Compose application using host networking. Set KAFKA_BOOTSTRAP to Kafka's host-reachable advertised listener (default: localhost:9092). Compose cannot resolve depends_on across separate invocations.

Examples

• "Create a custom RTVI CV microservice that runs YOLO26 in DeepStream and publishes object metadata so VSS Search and Alerts can consume it."
• "Validate the generated RTVI VSS service scaffold locally, then explain how to verify the live integration on a GPU host with a deployed VSS stack."
• "I do not have a trained YOLO26 ONNX, labels file, or parser .so yet. Can the live VSS integration still run?"

VSS source location

This skill ships the scaffold generator, but not the VSS deployment. Clone the public VSS Blueprint repository separately or reuse an existing checkout, then set VSS_ROOT to that checkout explicitly. Do not search for or clone VSS relative to the generated service directory:

# Customer-specific path to an existing VSS v3.2.1-compatible checkout.
export VSS_ROOT=/absolute/path/to/video-search-and-summarization
VSS_DEPLOY_DIR="${VSS_ROOT}/deploy/docker"
test -f "${VSS_DEPLOY_DIR}/compose.yml" || {
  echo "Missing VSS compose file: ${VSS_DEPLOY_DIR}/compose.yml" >&2
  exit 1
}
echo "Using VSS deployment: ${VSS_DEPLOY_DIR}"

Relevant customer-accessible VSS locations include:

• deploy/docker/scripts/dev-profile.sh — brings up Search or Alerts using the documented Quickstart flow
• deploy/docker/developer-profiles/dev-profile-search/
• deploy/docker/developer-profiles/dev-profile-alerts/
• deploy/docker/services/rtvi/rtvi-cv/ — stock RTVI-CV service definition

The custom service generated by this skill remains in <target-dir> and runs as a separate, host-networked Compose application alongside VSS. Set KAFKA_BOOTSTRAP to Kafka's host-reachable advertised listener (default: localhost:9092). Do not look for generated YOLO26 files inside either the DeepStream or stock VSS checkout.

Required reads

Read these before generating code so the implementation matches the documented VSS data path rather than a generic Kafka producer:

1. references/integration-contract.md — the exact mdx-raw protobuf contract. For segmentation/frame-mask services, read its segmentation frame-mask payload section before implementing msgconv, wrapper, Kafka, or protobuf changes.
2. references/yolo26-deepstream.md — DeepStream nvinfer config for YOLO26.
3. references/vss-profile-integration.md — how the service plugs into Search / Alerts / Behavior Analytics.
4. VSS Object Detection and Tracking — https://docs.nvidia.com/vss/latest/object-detection-tracking.html
5. VSS Behavior Analytics — https://docs.nvidia.com/vss/latest/behavior-analytics.html
6. VSS Search Workflow — https://docs.nvidia.com/vss/latest/agent-workflow-search.html
7. DeepStream Gst-nvmsgconv — https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgconv.html
8. DeepStream Gst-nvmsgbroker — https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgbroker.html

Inputs to confirm

Ask the user for these. Use placeholders if not provided — do not block.

• service-name (slug; becomes container name and python package)
• yolo26-onnx-path (host path to the YOLO26 ONNX export — not .pt)
• yolo26-labels-path (host path to a one-class-per-line labels file)
• yolo26-parser-lib (host path to the YOLO26 custom-parser .so, or the parser function name if compiled into the customer image)
• num-classes (integer; must match the labels file)
• target-vss-profiles — any subset of:
  • bp_developer_search_2d (Search Profile)
  • bp_developer_alerts_2d_cv (Alerts Profile + Behavior Analytics consumer)
• kafka-bootstrap (Kafka's host-reachable advertised listener; default localhost:9092)
• kafka-topic (default mdx-raw — VSS consumers expect this name)
• input-rtsp-uri or input video file path

Workflow

1. Read references/integration-contract.md. The fixed parts of the contract (topic name, payload type, schema library) are not negotiable if the goal is to plug into an existing VSS deployment. For segmentation/frame-mask services, follow the segmentation payload contract in that reference before changing msgconv or wrapper code.

2. Scaffold the YOLO26 reference service (do not generalize pgie paths unless the user explicitly needs another detector):

   python3 scripts/scaffold_rtvi_vss_service.py \
     --service-name <service-name> \
     --output-dir <target-dir> \
     --num-classes <N> \
     --vss-profiles bp_developer_search_2d bp_developer_alerts_2d_cv
   

3. Drop the customer's YOLO26 ONNX, labels file, and custom-parser library into the paths the generated service_config.json references. If those artifacts do not exist yet, stop after scaffolding and host-only unit tests; the live DeepStream service cannot start correctly without them.

4. Build the image:

   docker build -t <service-name>:dev <target-dir>
   

5. From the separate VSS checkout, bring up the matching Search or Alerts profile using the VSS Quickstart and deploy/docker/scripts/dev-profile.sh. Confirm that VSS and its one-shot Kafka topic initializer are ready, then start the generated service as a separate Compose application:

   For Search or Alerts, find the matching one-shot initializer without assuming a Compose project or container name:

   docker ps -a \
     --filter label=com.docker.compose.service=kafka-topic-init-container \
     --format 'table {{.Names}}\t{{.Status}}'
   

   Identify the selected deployment's initializer and confirm it is Exited (0), then start the generated service with the matching gate:

   cd <target-dir>
   docker compose -f compose/service.compose.yml \
                  --profile bp_developer_search_2d up -d
   
   # Or for Alerts:
   docker compose -f compose/service.compose.yml \
                  --profile bp_developer_alerts_2d_cv up -d
   

   Do not add a cross-file depends_on entry. The generated service uses network_mode: host. Set KAFKA_BOOTSTRAP to Kafka's host-reachable advertised listener (default: localhost:9092). The readiness check enforces startup order.

6. Verify the metadata flow:

   cd <target-dir>
   python3 tools/kafka_smoketest.py --describe-only
   python3 tools/kafka_smoketest.py --timeout 120
   

   The smoke test auto-detects Kafka only when exactly one matching container is running. If multiple deployments are active, pass the selected running name with --kafka-container. Its --bootstrap-server is resolved inside that Kafka container and normally remains localhost:9092, even when the detector's host-facing KAFKA_BOOTSTRAP uses a different port.

   A successful consume confirms at least one non-empty message arrived on mdx-raw within the timeout. It does not decode the protobuf payload or validate sensorId, objects, or bbox fields.

7. Confirm downstream pickup:

   • Search Profile: query the Video Analytics API for ingested events, run an embed query and an attribute query against the search workflow.
   • Alerts Profile: watch mdx-incidents for behavior-analytics output; the alert-bridge service should generate VLM-verified incidents.
   • Behavior Analytics: the selected VSS Search or Alerts deployment starts its own behavior consumer. This is part of the VSS stack, not another profile for the generated service. It consumes mdx-raw and emits behavior windows to mdx-incidents.

Service shape (generated)

The scaffolder emits this layout:

<service-name>/
├── Dockerfile                            DeepStream 9.1 + custom parser hook
├── README.md                             customer-facing build/run/plug-in guide
├── service_config.json                   declarative config (topic, profile flags, paths)
├── compose/
│   └── service.compose.yml               service def with profiles for VSS plug-in
├── pipeline/
│   ├── ds-app-config.txt                 deepstream-test5 derived; msgconv→msgbroker→mdx-raw
│   ├── ds-start.sh                       container entrypoint
│   └── configs/
│       ├── pgie-yolo26-config.txt        nvinfer config skeleton for YOLO26
│       ├── tracker-nvdcf.yml             NvDCF tracker config
│       ├── cfg_kafka.txt                 librdkafka producer overrides
│       ├── msgconv_config.txt            mega2d sensor context
│       └── labels.txt                    class labels placeholder
├── app/
│   ├── __init__.py
│   ├── contracts.py                      NvDsEventMsgMeta-aligned event envelope
│   ├── pipeline_plan.py                  declarative stage list (used by tests)
│   └── service.py                        adapter helpers for extension/derived events
├── tools/
│   └── kafka_smoketest.py                consumes one mdx-raw message and asserts it is non-empty
└── tests/
    ├── test_pipeline_config.py           asserts the Kafka adapter contract and msgconv→msgbroker→mdx-raw wiring
    ├── test_service.py                   exercises the python adapter without GPU
    └── test_compose.py                   asserts profile gates, host networking, and liveness healthcheck

Non-negotiable contract elements

The customer can change almost everything except these — they're what the VSS stack consumes:

• Kafka topic mdx-raw, using the host-reachable advertised listener selected by KAFKA_BOOTSTRAP (default: localhost:9092).
• msg-conv-payload-type=2 (NVDS_PAYLOAD_DEEPSTREAM_PROTOBUF) with msg-conv-msg2p-new-api=1 so the generated deepstream-app serializes frame/object metadata directly: protobuf serialized by msg-conv-msg2p-lib (libnvds_msgconv_mega2d.so, or libnvds_msgconv.so on DGX-SPARK/THOR); libnvds_kafka_proto.so is the msg-broker-proto-lib Kafka transport adapter only.
• A protobuf Frame payload with sensorId, timestamp, and objects[] carrying id, bbox, type, confidence. This is what vss-search-analytics-*, vss-behavior-analytics-*, and vss-video-analytics-api-* deserialize.
• network_mode: host.
• An explicit deployment precondition that the VSS kafka-topic-init-container completed successfully before this separate Compose application starts.

If any of these change, the customer is no longer plugging into VSS — they are running an isolated CV service.

Validation

Run unit tests on any host (no GPU required):

cd <target-dir>
python3 -m unittest discover -v -s tests -p 'test_*.py'

Run end-to-end on a host with GPU + a deployed VSS stack:

cd <target-dir>
python3 tools/kafka_smoketest.py --describe-only
python3 tools/kafka_smoketest.py --timeout 120

A successful smoke test confirms non-empty bytes arrived on mdx-raw within the timeout. It does not decode the protobuf payload or prove the message came from this service specifically. To verify downstream pickup, check the Search / Alerts / Behavior Analytics services via their own APIs.

Do not present local unit-test success as proof that live VSS integration is ready. Without the customer ONNX, labels file, and parser library, the live DeepStream path remains blocked even if scaffolding and unit tests succeed.

Implementation guardrails

• Do not invent topic names. mdx-raw is the single perception ingress. Per-service prefixes break the existing VSS consumers.
• Do not switch to msg-conv-payload-type=0 or =1. VSS deserializers expect type 2.
• Do not bridge through a custom Python Kafka producer. The DeepStream nvmsgconv → nvmsgbroker boundary is what produces correctly-framed protobuf with timestamps in nanoseconds. A Python producer drift will silently degrade Behavior Analytics.
• Keep track_id stable across frames. Behavior Analytics derives dwell and direction from track continuity; a per-frame regenerated id makes every frame look like a new object.
• Do not embed credentials, NGC tokens, or absolute customer paths in the scaffolded files. The generated tree must be portable.