add-middleware

Add a Middleware Type

Companion Guidance

Use karpathy-guidelines alongside this skill for implementation or review work. Keep changes scoped, surface assumptions, and define focused validation before editing.

NeMo Relay supports guardrails (validate/gate) and intercepts (transform) at various pipeline stages. Adding a new middleware type requires changes across all layers.

Use this skill when introducing a new middleware registration surface or adding middleware behavior to a new pipeline stage.

Lock The Design First

Decide these before editing code:

• Is this for tools, LLMs, marks, scope events, or a combination?
• Is it a conditional guardrail, sanitize guardrail, request intercept, or execution intercept?
• Does it run on request input, inner callable execution, stream chunks, or final response output?
• Is the callback fallible, and how should callback failures propagate?
• Does it need both global and scope-local registration?
• What should subscribers and exporters observe in the event payload after this middleware runs?
• If this is an event sanitizer, which of data, category_profile, and metadata can change, and is the event used only as immutable context?

Pipeline Order

Refer to docs/about-nemo-relay/concepts/middleware.mdx for the full diagrams.

• Tool execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
• LLM execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
• Mark and scope events: specialized tool or LLM sanitizer (when applicable) -> mark or scope event sanitizer -> subscriber and exporter dispatch

Tool execution callbacks and each execution-intercept next continuation return the canonical ToolExecutionResult { result, annotation }. A forwarding intercept must preserve both fields in ToolExecutionInterceptOutcome; Relay retains pending_marks separately. Tool sanitize-response guardrails receive only result. Scope-end event sanitizers govern the annotation after Relay projects it to category_profile.tool_result_annotation.

Core Steps

1. Define or reuse the callback type alias in crates/core/src/api/runtime/callbacks.rs.

pub type MyNewFn = Box<dyn Fn(&str, Json) -> Json + Send + Sync>;

2. Add the registry field to NemoRelayContextState in crates/core/src/api/runtime/state.rs.

Add a SortedRegistry<GuardrailEntry<MyNewFn>> or SortedRegistry<Intercept<MyNewFn>> field to the state struct.

3. Add registration and deregistration APIs in crates/core/src/api/.

Use the existing global_*_registry_api! and scope_*_registry_api! macro patterns in crates/core/src/api/registry.rs. Both global and scope-local variants are needed unless the design explicitly rules one out.

4. Add chain execution helpers to NemoRelayContextState in crates/core/src/api/runtime/state.rs.

Follow the pattern of tool_sanitize_request_chain or tool_request_intercepts_chain.

5. Wire the chain into the execute path.

Update the relevant lifecycle owner to call the new chain method at the appropriate pipeline stage. Tool and LLM paths live in crates/core/src/api/tool.rs and crates/core/src/api/llm.rs; shared mark and scope event sanitization lives in crates/core/src/api/shared.rs and is called from crates/core/src/api/scope.rs.

6. Expose the new middleware surface in every affected binding.

Follow the add-binding-feature skill for the cross-binding implementation checklist.

Required Tests

• Registration and duplicate-name behavior
• Deregistration and no-op missing-name behavior
• Ordering by priority
• Callback failure policy, including fail-open behavior when required
• Scope-local registration, inheritance, and cleanup on pop
• Event payload semantics after middleware mutation
• Tool execution result and annotation preservation, replacement, and removal when the middleware touches tool execution
• Mark and scope event field semantics, including immutable identity fields
• Parity coverage in every affected binding

Key References

• Pipeline logic: crates/core/src/api/tool.rs, crates/core/src/api/llm.rs
• Type aliases: crates/core/src/api/runtime/callbacks.rs
• Runtime state and chain builders: crates/core/src/api/runtime/state.rs
• Scope-local registry merging: crates/core/src/context/registries.rs
• Registry: crates/core/src/registry.rs
• Pipeline docs: docs/about-nemo-relay/concepts/middleware.mdx
• Architecture docs: docs/about-nemo-relay/architecture.mdx
• Registration examples: docs/instrument-applications/advanced-guide.mdx
• Validation: validate-change