deepstream-sop

Use this skill when building, deploying, evaluating, debugging, or measuring latency for the DeepStream SOP Inference Microservice — a GPU-accelerated FastAPI…

DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice — a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance monitoring on industrial video feeds.

Reference repository: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp Local reference code: sop-inference-bp/ directory (from a local clone of the repository)

Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

Stage	Role	Model class	Default	Swap via
Stage 1 (CV)	Per-frame boundary scoring → chunk segmentation	Generic Event Boundary Detection (GEBD)	DDM (MCG-NJU/DDM) via Triton Python backend	Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5)
Stage 3 (VLM)	Per-chunk action classification	Vision-language model via vLLM	Cosmos Reason 1 7B (Reason 2 also supported)	Set `VLLM_MODEL_PATH` to a different HF ID or local path

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

Chunking is selectable per request (§ 2): default ddm-net uses GEBD; uniform produces fixed-length chunks and bypasses Stage-1 GEBD (§ 3, § 6). DDM temporal window is configurable via FRAMES_PER_SIDE / SEQUENCE_BATCH (§ 4, § 5), with optional TensorRT (§ 5).

Architecture Overview

Runs in a Docker container (nvds-action-sop) alongside a Kafka container. Full diagram: references/sop_architecture.svg.

Data flow through the 4-stage SOPVideoProcessor pipeline (per-request):

Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │              │ chunk results                    │
                                   │              ▼                                 │
                                   │         final_queue                            │
                                   └────────────────────────────────────────────────┘
                                          │
Output                                    ▼
──────                             ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages   │
Non-streaming (chat.completion)    │ (JSON/Protobuf)  │
Prometheus metrics (/v1/metrics)   └────────┬────────┘
                                            ▼
                                   Docker Container: kafka
                                   (apache/kafka:3.7.0)

Section Index

Each section is a standalone file in references/ — load only what your task needs.

§	File	Responsibility
1	`skill_01_fastapi_endpoints.md`	FastAPI endpoints, server init, Prometheus metrics
2	`skill_02_pydantic_schemas.md`	Request/response Pydantic models (`api_types.py`)
3	`skill_03_deepstream_pipeline.md`	DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline
4	`skill_04_config_templates.md`	nvdspreprocess / nvinferserver config templates + rendering
5	`skill_05_triton_ddm_model.md`	Triton model repo, config.pbtxt, model.py, ddm_net.py
5b	`skill_05b_custom_postprocess.md`	C++ postprocess plugin, Makefile, IOptions API
6	`skill_06_sop_process_manager.md`	SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka
6b	`skill_06b_sop_checker.md`	SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response
7	`skill_07_sse_streaming.md`	SSE generator, stream response formatting, dummy test mode
8	`skill_08_basler_camera.md`	Basler camera support, Pylon SDK, emulation, formats
9	`skill_09_docker_build_deploy.md`	Docker build, deploy, .env configuration
10	`skill_10_test_suite.md`	Test suite coverage, assertions, running tests
11	`skill_11_env_variables.md`	All environment variables reference
12	`skill_12_evaluation_workflow.md`	End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report
13	`skill_13_verification_curl.md`	Verification steps and curl examples
14	`skill_14_implementation_checklist.md`	Implementation checklist: file copy list, generated files, Docker prereqs, verification
15	`skill_15_latency_measurement.md`	TTFC and C2C latency measurement for file input via SSE streaming
16	`skill_16_message_schema.md`	Kafka message schema selection (`JSON` default vs `NvProtoSchema`) and extending messages with custom data
17	`skill_17_camera_latency_measurement.md`	Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps
18	`skill_18_rtsp_streaming_output.md`	OPT-IN RTSP streaming output: `tee1`-tap re-stream, `RTSPStreamingServer`, `SW_ENCODER` toggle. Generate only when user explicitly requests RTSP

For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.

§ 18 is opt-in — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the RTSP_* rules below.

Key Files Map

The full source-to-target file mapping lives in skill_14_implementation_checklist.md:

Files copied verbatim from references/ (non-trivial algorithms — cycle detection, qwen_vl_utils preprocessing, DeepStream IOptions API, protobuf sources) with the rationale per file.
Files copied as adaptable templates (Dockerfile, compose.yaml, Triton config and model.py, ddm_net.py, Pylon emulation config, etc.).
Files generated from skill sections — each annotated with the Critical Rules below that the generation must follow exactly.
Docker build prerequisites and post-build verification checklist.

Config files (nvds_preprocess_template.txt, nvds_inference_template.txt, vlm_prompts.txt) are used as-is from configs/.

When skill_06b is loaded, read configs/actions.json from the project root and run the § 6b-G generation workflow to produce nvds_action_detector/missing_number_detector.py. If configs/actions.json is absent or invalid, fall back to copying the reference file.

Critical Rules

Each rule's full detail lives in the linked skill_NN_*.md reference file.

Tag	Rule summary	Details in
`MANAGER_INIT_IN_MAIN`	`SOPProcessManager` init in `main()` before `uvicorn.run()` — not inside `lifespan()`	`skill_01_fastapi_endpoints.md`
`NAMED_KWARGS`	`create_video_processor()` uses named kwargs; camera args as separate kwargs	`skill_06_sop_process_manager.md`
`LIVE_REQUIRES_STREAM_TRUE`	`stream: true` required for live inputs (RTSP / camera)	`skill_08_basler_camera.md`
`VLM_DISABLED_DISABLES_SOP_CHECKER`	`DISABLE_VLM_INFERENCE=true` auto-disables SOP checker at import	`skill_06_sop_process_manager.md`
`CHUNK_PARAMS_MAX_LENGTH`	`ChunkParams.max_length_sec` = 10s internal; 60s API default	`skill_06_sop_process_manager.md`
`VLM_WARMUP_BEFORE_DDM`	`ModelInitializer`: VLM warmup FIRST, then CV dummy pipeline	`skill_06_sop_process_manager.md`
`VLM_WARMUP_3_FRAMES`	VLM warmup needs 3 frames (`torch.zeros`) — Qwen3VL hangs on < 3	`skill_06_sop_process_manager.md`
`THREAD_POOL_SIZES`	4 thread pools: `cv`(32), `clip`(32), `vlm_inference`(64), `vlm_request`(64)	`skill_06_sop_process_manager.md`
`MEDIA_INFO_PYMEDIAINFO`	Media info via `pymediainfo`; live sources set fps=30/duration=inf directly	`skill_06_sop_process_manager.md`
`CAMERA_EMULATION_PYLON_CAMEMU`	`PYLON_CAMEMU=1` for camera emulation (serial `0815-0000`)	`skill_08_basler_camera.md`
`DEEPSTREAM_LIB_HIDE`	DeepStream lib hide trick: rename lib → lib.tmp during gst-plugin-pylon build	`skill_08_basler_camera.md`
`VLM_REAL_GPU_FRAMES`	VLM uses real GPU frames via `DecodedFrameRetriever`; never `torch.zeros` for inference	`skill_06_sop_process_manager.md`
`BUFFER_RETRIEVER_STATIC_BASE`	`DecodedFrameRetriever` MUST inherit `BufferRetriever` statically via `super().__init__()`; runtime `__class__.__bases__` mutation hangs `pipeline.attach()`	`skill_06_sop_process_manager.md`
`FRAME_RETRIEVER_PRIORITY`	`create_inference_pipeline`: `frame_retriever=` kwarg takes priority over `frame_queue`	`skill_03_deepstream_pipeline.md`
`MUX_ORIGINAL_RESOLUTION`	nvstreammux uses original resolution (not 224); pass `mux_width/mux_height` from `get_media_info()` (probe live RTSP for non-camera inputs; camera path unaffected)	`skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md`
`FILE_URI_NO_DOUBLE_PREFIX`	`create_inference_pipeline` file source: check `file_path.startswith("file://")` before prepending — API passes `file://` URLs directly	`skill_03_deepstream_pipeline.md`
`CLEANUP_ON_DISCONNECT`	Pipeline cleanup on client disconnect via `trigger_stop_processors` in `try/finally`	`skill_07_sse_streaming.md`
`UNIFIED_CLIP_POST_PROCESS`	Unified `clip_post_process()` for file + live; `stop()` puts `None` in `_score_queue`	`skill_06_sop_process_manager.md`
`ABORT_INFLIGHT_VLM`	Abort in-flight VLM requests on `stop()` via `llm.abort(req_id)`	`skill_06_sop_process_manager.md`
`LOGGER_EXPORT_GET_LOGGER`	`ds_logger.py` must export `get_logger`	`skill_06_sop_process_manager.md`
`KAFKA_USE_CREATE_PRODUCER`	Kafka: use `create_producer()` from `messager.py`; no `Messager` class	`skill_06_sop_process_manager.md`
`USER_PROMPT_PRIORITY`	User request text takes priority over `VLM_PROMPT_PATH` file; `{"type":"text"}` in the request overrides the config-file prompt	`skill_06_sop_process_manager.md`
`EVAL_USE_CONFIG_PROMPT`	Eval/latency requests omit request text by default so the VLM uses `VLM_PROMPT_PATH`	`skill_12_evaluation_workflow.md`, `skill_13_verification_curl.md`, `skill_15_latency_measurement.md`, `skill_17_camera_latency_measurement.md`
`CHUNK_SCHEMA_FIELD_NAMES`	Chunk schema: `chunk_idx`, `cv_boundary_score`, `checker_result`; summary `chunk_idx=-1`	`skill_06_sop_process_manager.md`
`SEQUENTIAL_FRAME_DRAIN`	Drain `decoded_frame_queue` (FIFO, shared across chunks) in a SINGLE thread and submit VLM per chunk incrementally; parallel drain steals frames → 0-frame chunks / wrong VLM input	`skill_06_sop_process_manager.md`
`WALL_CLOCK_BEFORE_GPU`	`DecodedFrameRetriever.consume()`: capture `wall_clock_entry = time.time()` BEFORE GPU dlpack; queue 3-tuple `(timestamp, wall_clock_entry, tensor)`	`skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md`
`CHUNK_E2E_PIPELINE_TIMESTAMPS`	Write `pipeline_chunk_end_timestamp` (last frame wall_clock) and `pipeline_vlm_ready_timestamp` (`tm_e2e.now()`) into `chunk_info` for camera latency (§ 17)	`skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md`
`VLM_INFERENCE_REQUIRED_KWARGS`	Every `VLLMInference.inference()` call must pass `video_fps`, `system_prompt`, `max_completion_tokens`	`skill_06_sop_process_manager.md`
`UNIFORM_CHUNKING_BYPASSES_DDM`	`chunking_options.algorithm="uniform"` → fixed-length chunks; `create_inference_pipeline(uniform_chunk=True)` skips DDM but keeps `tee1` fanout; Stage 2 uses `uniform_clip_post_process`	`skill_02_pydantic_schemas.md`, `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md`
`DDM_TEMPORAL_CONFIGURABLE`	`SLIDING_WINDOWS_SIZE = 2*FRAMES_PER_SIDE + SEQUENCE_BATCH` rendered into preprocess/nvinferserver (no hard-coded 18); Triton `config.pbtxt` sequence dim `-1`	`skill_04_config_templates.md`, `skill_05_triton_ddm_model.md`
`DDM_TRT_OPTIONAL_PATH`	`DDM_TRT_OPTIMIZATION=true` runs DDM via TensorRT (per-thread contexts, fixed batch = SEQUENCE_BATCH); PyTorch fallback; never both. PyTorch is default	`skill_05_triton_ddm_model.md`
`DDM_TRT_STREAM_ORDERING`	`DDMTensorRTEngine.infer()`: `wait_stream(current)` → `execute_async_v3` → `torch.cuda.synchronize(device)` (NOT per-stream). Per-stream sync leaves TRT aux-stream work in flight → gst-CV SIGSEGV (NVBug 6289256)	`skill_05_triton_ddm_model.md`
`METADATA_LICENSE_FROM_FILE`	`/v1/metadata` reads `licenseInfo` from `DS_SOP_LICENSE_PATH` (default `/opt/nvidia/nvds_sop/license.txt`); never hard-code license text	`skill_01_fastapi_endpoints.md`
`CAMERA_EMULATION_FRAMES_RGB`	Pylon emulation PNGs must be explicit 3-channel RGB (matches `Emulation_0815-0000.pfs PixelFormat=RGB8Packed`); generate via `nvvideoconvert ! videoconvert ! "video/x-raw,format=RGB" ! pngenc`	`skill_08_basler_camera.md`
`COMPOSE_ENV_PASSTHROUGH`	`docker compose` only substitutes `${VAR}` references; every runtime env var must be explicitly listed under `environment:` to reach the container.	`skill_09_docker_build_deploy.md`

The four RTSP_* rules below apply only when the optional RTSP streaming-output feature (§ 18) is requested. They do not apply to the default build — skip them if the user did not ask for RTSP output.

| RTSP_OUTPUT_TAPS_TEE1 | RTSP output branch links from the existing tee1 (added after the main inference link) only when rtsp_port is present. | skill_18_rtsp_streaming_output.md | | RTSP_LEAKY_QUEUE_TINY | RTSP branch queue must be leaky=2 + tiny cap (max-size-buffers=2) to prevent backpressure and NVMM pool exhaustion. | skill_18_rtsp_streaming_output.md | | RTSP_KEYINT_MAX_30 | RTSP H.264 encoder must set key-int-max=30 (and B-frames disabled) to allow downstream seeking. | skill_18_rtsp_streaming_output.md | | RTSP_ENCODER_FALLBACK | Select software/hardware H.264 encoder based on SW_ENCODER with MJPEG fallback. | skill_18_rtsp_streaming_output.md |