nvrx-attr

par nvidia

Couche d'orchestration au-dessus des modules d'attribution de nvidia_resiliency_ext. Fournit une analyse des logs, une analyse fr, et un retour d'injection de fautes orienté Megatron-LM…

npx skills add https://github.com/nvidia/nvidia-resiliency-ext --skill nvrx-attr

Télécharger ZIP GitHub

338

Attribution Skills

High-level orchestration layer over the nvidia_resiliency_ext.attribution modules. Each subdirectory is a self-contained skill with its own SKILL.md and helper scripts.

Skills

Directory	Purpose	Entry point
`log-analysis/`	Analyze SLURM job logs for failure root-cause and restart decisions	`RestartAgent` (`python -m nvidia_resiliency_ext.attribution.restart_agent.cli`)
`fr-analysis/`	Analyze NCCL flight-recorder dumps for collective-hang root-cause	`CollectiveAnalyzer` (`fr_attribution.py`)
`fault-injection-loop/`	Run a batched SLURM fault-injection feedback loop and score attribution accuracy	`prepare_node_alloc.sh` / `watch_and_analyze.sh`

How skills relate to the library

src/nvidia_resiliency_ext/
├── attribution/
│   ├── restart_agent/                    ← log-analysis implementation
│   ├── trace_analyzer/fr_attribution.py  ← fr-analysis implementation
│   └── mcp_integration/                  ← packaged restart-agent + FR MCP tools
└── skills/
    └── nvrx-attr/                        ← this skill bundle
        ├── log-analysis/
        ├── fr-analysis/
        └── fault-injection-loop/

Use the packaged restart_agent and trace_analyzer entry points for current log and FR analysis. Legacy LogSage, SPLITLOG, and combined LogSage+FR tools live under attribution/legacy_logsage/ for source-checkout workflows only.

Common prerequisites

LLM_API_KEY environment variable, LLM_API_KEY_FILE, or ~/.llm_api_key
Source checkout or installed package with the attribution extra for restart-agent and FR analysis
The fault-injection loop has only been validated with Megatron-LM training scripts

Fault-Loop Local Setup

Before using fault-injection-loop/, create the local config file from the tracked template and fill in your site-specific values:

cp scripts/user.env.example scripts/user.env

The feedback-loop scripts require src/nvidia_resiliency_ext/skills/nvrx-attr/scripts/user.env to exist at runtime. Keep user.env local and untracked.