nemo-relay-tune-performance

por nvidia

Plan a measured NeMo Relay adaptive tuning rollout after baseline scopes, tool calls, LLM calls, and observability are working; use this skill to improve…

npx skills add https://github.com/nvidia/nemo-relay --skill nemo-relay-tune-performance

Descargar ZIP GitHub

Tune Performance With Adaptive Behavior

Use This When

Use this skill when a user has baseline NeMo Relay instrumentation and wants to improve latency, parallelism, prompt-cache behavior, or model-request behavior from runtime signals.

Do Not Use This When

Do not use this skill when the application is not instrumented yet. Start with nemo-relay-instrument-calls or nemo-relay-start first.

Default Guidance

Observe first, compare against a baseline, then enable one behavior change at a time.
Use the adaptive plugin component rather than inventing separate tuning logic or hand-registering adaptive behavior at every call site.
Start with in-memory state and telemetry-only behavior for local development.
Move to persistent state only when learned signals must survive restarts or be shared across workers.
Add active behavior only after representative runtime events show what should change.

Embedded Adaptive Model

Adaptive behavior is configured through the first-party plugin component with kind adaptive.
Adaptive requires existing NeMo Relay scopes, managed tool or LLM calls, and lifecycle events because it learns from runtime signals.
Main configuration areas are state, telemetry, adaptive hints, tool parallelism, Adaptive Cache Governor (ACG), and rollout policy.
State backends are in_memory and redis.
Tool-parallelism modes are observe_only, inject_hints, and schedule.
Adaptive Cache Governor providers are passthrough, anthropic, and openai; omit ACG until prompt-cache planning is needed.
Helper APIs exist in Rust nemo_relay_adaptive, Python nemo_relay.adaptive, and Node.js nemo-relay-node/adaptive. Go and raw FFI are source-first or advanced surfaces.

Default Path

Confirm the app already emits expected scope, tool, and LLM events.
Capture a baseline for the workflow you want to improve.
Enable adaptive telemetry with in-memory state.
Run representative traffic and inspect reports or runtime events.
Choose one tuning surface: hints, tool parallelism, or ACG.
Enable the smallest behavior change in config.
Compare results against the baseline and keep a rollback path.

Failure Modes To Avoid

Do not enable scheduling before tool idempotency and race behavior are known.
Do not enable prompt-cache planning before provider payloads are stable.
Do not treat adaptive hints as mandatory instructions unless the consuming path explicitly defines that contract.
Do not use environment variables as the primary adaptive configuration model.
Do not tune from a single run or unrepresentative traffic.

Use Another Skill When

You need the exact adaptive config shape -> nemo-relay-tune-adaptive-config
You need to consume adaptive hints or scheduling guidance in app logic -> nemo-relay-tune-adaptive-hints
You need to build reusable plugin behavior instead of configuring the built-in adaptive component -> nemo-relay-build-plugin