simready-foundation-add-runtime-test

SimReady Add Runtime Test

Purpose

Use this skill when static validators are not enough to prove that a feature or profile works in a runtime, and the runtime evidence must live in the SimReady Foundation repository. Runtime tests are authored in the owning tier's importable <test-package> and executed through the simready-benchmark plan, run, stamp, and report pipeline. The core tier currently uses simready_benchmark_kit_suite; other tiers may publish a different import package and must declare it through their descriptor's runtime_tests_path.

This skill is deliberately scoped to the Foundation-repository wiring. The generic mechanics of authoring, packaging, running, and reporting benchmark tests are owned by the simready-benchmark library's own agent skills; do not duplicate them here. Use those skills for the test-authoring and pipeline mechanics, and use this skill for how a test is placed into this repository and kept in sync with the Foundation docs, feature manifests, and profiles.

Do not hand-edit generated plans, result JSON, or report artifacts. Change the Python test source, its documentation, and focused tests instead.

Delegate Mechanics to the simready-benchmark Library Skills

The external simready-benchmark library ships its own agent skills that are the source of truth for the framework mechanics. Locate them with simready-benchmark --skills-path, then read and follow the relevant one:

• write-test-pack - the @test decorator, required metadata (features, name, description, expected_video, version, engine, config_defaults, max_duration), RunContext usage, media roles, config-driven tests, and packaging/discovery through the simready_benchmark.tests entry-point group.
• run-tests - plan, run, and report a suite against a profile.
• diagnose-failures - read a report and propose a scoped re-run.

Treat the installed library skills as authoritative for API names and flags; they match the installed framework version. This skill only adds what those skills do not know: the Foundation repository's layout, docs, and spec bindings.

What This Skill Owns (Foundation-Specific)

• Deciding whether the test belongs to a Foundation tier. A test that proves a tier-owned feature should ship with that tier. An independent OEM or experimental test pack that does not own the feature should use the Benchmark write-test-pack skill and either an independent wheel or an explicit development path.
• Placing a tier-owned test directly under nv_core/tiers/<tier-directory>/<test-package>/fet###_* and focused unit tests under nv_core/tiers/<tier-directory>/tests/runtime_tests/unit/. The package name comes from the owning tier descriptor; it is not globally fixed.
• Keeping the tier wheel's package list and its simready.tier descriptor's runtime_tests_path synchronized. The descriptor points directly at the importable test package directory; it does not point at a runtime_tests container.
• Keeping the canonical FET_###_<RUNTIME> decorator entry and semantic version constraint synchronized with the owning feature manifest and profile TOMLs. The decorator is the executable feature-to-test binding; feature manifests do not carry a separate runtime_tests selector. The current @test decorator has no requirement argument, so document the requirement or behavioral contract in the test description and matching Foundation docs.
• Updating the matching documentation under the owning <test-package>/docs/ directory and any feature/profile acceptance text that promises this runtime evidence. The documentation assembler publishes the tier-declared source at the stable guides/benchmark/tests/ URL.

Prerequisites

Before editing, read:

• AGENTS.md
• nv_core/sr_specs/docs/guides/benchmark/benchmark.md
• nv_core/sr_specs/docs/guides/benchmark/overview.md
• nv_core/sr_specs/docs/guides/benchmark/pipeline.md
• nv_core/sr_specs/docs/guides/benchmark/running.md
• nv_core/sr_specs/docs/guides/benchmark/reading-reports.md
• nv_core/tiers/<tier-directory>/<test-package>/docs/tests.md (or the landing page declared by that test package)
• nv_core/tiers/<tier-directory>/<test-package>/docs/authoring.md
• nv_core/tiers/<tier-directory>/pyproject.toml
• nv_core/tiers/<tier-directory>/simready/foundation/<tier-module>/_tier.py
• nv_core/tiers/<tier-directory>/<test-package>/
• nv_core/tiers/<tier-directory>/tests/runtime_tests/unit/
• relevant feature/profile docs that need runtime evidence
• the neighboring runtime tests for the same feature family

Inputs

Collect or infer:

Instructions

Use this checklist when changing the repository. It covers the Foundation-specific wiring only; follow the simready-benchmark library skills (see above) for the @test authoring, packaging, and pipeline mechanics.

1. Decide whether runtime evidence is required by the feature/profile or only useful as supplementary confidence.
2. Read the matching feature manifest and feature documentation. Identify the canonical FET_###_<RUNTIME> decorator entry and semantic version constraint, then confirm the same feature version is selectable from the relevant profile TOMLs. The decorator is the executable feature-to-test binding; do not add a separate runtime_tests selector to the feature manifest. Identify the requirement or runtime promise it proves for documentation, but do not pass a requirement keyword to @test; that keyword is not part of the current API.
3. Read the simready-benchmark write-test-pack skill (via simready-benchmark --skills-path) and follow it for the @test authoring mechanics, decorator metadata, RunContext usage, and config defaults. Do not restate those mechanics here.
4. Choose the ownership model:
   • For a tier-owned feature, resolve <test-package> from the tier descriptor's runtime_tests_path, add the test to nv_core/tiers/<tier-directory>/<test-package>/fet###_*, and keep it in the same wheel as that tier.
   • For an independent published pack, follow the Benchmark skill and register the wheel under simready_benchmark.tests.
   • For an unpublished pack, point --tests-path at the importable package directory. Explicit paths are additive to installed tier and independent providers.
5. For a tier-owned test, inspect neighboring modules and add each independently reportable test in its own Python file. Reuse the suite's scene, context, cleanup, configuration, and unit-test patterns. Do not recreate a separate nv_core/runtime_tests or nv_core/testing_tools provider tree.
6. Confirm the owning tier's pyproject.toml includes both its simready package and <test-package>, and that the tier descriptor's runtime_tests_path resolves directly to that importable package directory. For the core tier, <test-package> is currently simready_benchmark_kit_suite. Descriptor import must remain dependency-light and must not import tests, Kit, Isaac, or pxr.
7. Bind the test to this repository's specs with the canonical Foundation feature ID and semantic version constraint in the decorator's features list. Confirm that ID/version against the owning feature manifest and relevant profile TOMLs. Provider identity is a discovery concern handled by the tier descriptor, simready_benchmark.tests entry point, or --tests-path; do not author provider/test selectors in feature manifests. Keep pure calculation and discovery logic outside the coroutine and add focused unit tests; add registration/metadata coverage when changing mappings or report behavior.
8. Update the matching page under the owning suite's docs/ directory and any feature/profile acceptance text that promises this runtime evidence. Keep the suite index/toctree and the tier's declarative documentation-source mapping synchronized. Do not create a second copy under sr_specs/docs.
9. Validate both delivery paths that changed:
   • Source checkout: use --foundations-path <repo> or point --tests-path directly at the test package while developing, then run --list-tests.
   • Tier wheel: build and install the tier wheel in a clean environment and run --show-config plus --list-tests without --tests-path. Then run a scoped --plan-only and --no-stamp execution. Inspect the report/media and summarize the pass/fail/skip signal. If Kit/Isaac is unavailable, report a blocked runtime check rather than claiming a pass.

Examples

Example request:

Add a runtime test proving a SimReady factory asset connection point imports and aligns correctly in Kit.

Expected result summary:

changed_files: new docs, manifests, indexes, or validation scaffolding
validation: discovery, plan-only, focused unit checks, and runtime evidence when available
remaining_gaps: requirement, validator, adapter, profile, or runtime-test follow-up

Policies

• Generated plans, result JSON, media, and report folders are artifacts, not source of truth.
• Runtime tests should be reproducible from the repository, installed Benchmark packages, engine configuration, asset path, and command line.
• Tier-owned tests ship in the tier wheel and are advertised only through the tier descriptor. Independent test-only wheels use simready_benchmark.tests.
• The decorator's canonical feature ID and semantic version constraint own the executable feature-to-test binding. Keep them synchronized with the tier's feature manifests and profile TOMLs.
• --tests-path must point at the importable package directory containing __init__.py, not at the repository root.
• Keep user/CI-owned executable paths and credentials out of committed tests and docs.
• Do not change an asset merely to make a runtime test pass unless the task explicitly includes asset conformance.
• If the runtime cannot run locally, still add clear commands, expected signals, and focused non-runtime validation.

Limitations

• Do not mutate published feature or profile versions in place.
• Do not invent requirement IDs or validator behavior when the contract is ambiguous; record the question.
• Do not skip index, manifest, validation, or downstream follow-up notes.

Troubleshooting

• Error: the new concept overlaps an existing artifact. Solution: update the existing capability, requirement, feature, profile, or adapter instead.
• Error: names or IDs conflict. Solution: re-check naming conventions and nearby indexes before editing further.
• Error: validation strategy is unclear. Solution: document deferred validation and the exact follow-up skill.

Resources

• assets/openai.yaml preserves optional UI metadata for clients that read skill display hints. It is not required for the workflow.

Summary Format

Report: