fault-injection-loop

閉環故障注入與歸因準確度基準測試。從優先級池中提取（故障類型、排名、迭代次數、節點）實驗，並提交2…

Skill: fault-injection-loop

Iterative closed-loop skill that runs a prioritized fault-injection experiment pool 2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates gaps across the matrix, and proposes targeted improvements to attribution modules.

Overview

┌───────────────────────────────────────────────────────────────────────┐
│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │
│               GPU faults first, then crash, Python-hang, signal       │
│                                                                        │
│  repeat until pool exhausted:                                          │
│  1. SUBMIT   → sbatch 2 jobs from pool head                            │
│  2. WAIT     → poll until both jobs leave RUNNING/PENDING              │
│                                                                        │
│  after all jobs done:                                                  │
│  3. ANALYZE  → watch_and_analyze.sh: /log-analysis + /fr-analysis     │
│               per completed job, streaming as jobs finish              │
│  4. SCORE    → compare attribution output vs injected ground truth     │
│  5. AGGREGATE→ build results table; identify systematic failure modes  │
│  6. IMPROVE  → patch log_analyzer/nvrx_logsage.py                     │
│  7. LOOP     → re-run same pool with updated attribution code          │
└───────────────────────────────────────────────────────────────────────┘

Step 0 — Fault Pool Design

The pool is defined as an ordered list of FAULT_TYPE:RANK:ITER:NODES entries inside scripts/prepare_node_alloc.sh. Default pool (34 experiments, 17 batches):

# GPU hangs — highest priority; full rank sweep across all node counts
GPU_SLEEP:1:5:2   GPU_SLEEP:0:5:2      # 2-node: rank-1, rank-0
GPU_SLEEP:4:5:2   GPU_SLEEP:7:5:2      # 2-node: mid-rank, last-rank
GPU_SLEEP:1:5:4   GPU_SLEEP:0:5:4      # 4-node: rank-1, rank-0
GPU_SLEEP:8:5:4   GPU_SLEEP:15:5:4     # 4-node: mid, last
GPU_SLEEP:1:5:8   GPU_SLEEP:0:5:8      # 8-node: rank-1, rank-0
GPU_SLEEP:16:5:8  GPU_SLEEP:31:5:8     # 8-node: mid, last

# GPU errors — high priority; rank-0 and rank-1 across all node counts
GPU_ERROR:1:5:2   GPU_ERROR:0:5:2
GPU_ERROR:1:5:4   GPU_ERROR:0:5:4
GPU_ERROR:1:5:8   GPU_ERROR:0:5:8

# Crash faults
SIGKILL:1:5:2     SIGKILL:0:5:2
SIGKILL:1:5:4     SIGKILL:1:5:8
SEGFAULT:1:5:2    SEGFAULT:0:5:2
SEGFAULT:1:5:4    OS_ABORT:1:5:2

# Python-level hangs
LOCK_GIL:1:5:2    LOCK_GIL:0:5:2
WORKLOAD_EXC:1:5:2  ASYNC_EXC:1:5:2

# Signals
SIGTERM:1:5:2     SIGINT:1:5:2
SIGSTOP:1:5:2     SIGNAL_EXC:1:5:2

Rank coverage per node count (4 GPUs/node):

Nodes	Total ranks	rank-1	mid	last
2	8	1	4	7
4	16	1	8	15
8	32	1	16	31

To run a custom subset, override POOL before calling the script:

POOL="GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2" bash scripts/prepare_node_alloc.sh

Local User Config

Start from the tracked template:

cp scripts/user.env.example scripts/user.env

Then edit scripts/user.env with cluster-specific settings. This file is sourced by run_session.sh, prepare_node_alloc.sh, watch_and_analyze.sh, l4_gb200_reduced.sh, and n3_super_gb200_fi.sh. It is required for this skill to run and is intended to stay local and untracked.

Variable	Default	Description
`WORKLOAD`	`llama4_scout`	Select a registered workload by name (see `scripts/workloads.conf`)
`ACCOUNT`	(cluster default or `scripts/user.env`)	SLURM account
`PARTITION`	(cluster default or `scripts/user.env`)	SLURM partition
`GPUS_PER_NODE`	`4`	GPUs per node
`TIME`	`00:30:00`	Per-job wall-clock limit
`BATCH_SIZE`	`2`	Jobs submitted per round
`POLL_INTERVAL`	`30`	Seconds between queue polls
`BASE_EXPERIMENTS_DIR`	`${HOME}/nvrx-attr-experiments`	Root for all output
`MEGATRON_REPO_HOST_PATH`	`${HOME}/megatron-lm-main`	Host path to the Megatron checkout mounted into the container
`SHARED_TMP_BASE_DIR`	`${HOME}/tmp`	Shared filesystem path used for cross-step coordination
`WORKSPACE_HOST_PATH`	`${HOME}/tmp`	Host path mounted at `/workspace` inside the container
`CONTAINER_IMAGE`	`nvcr.io/nvidia/nemo:26.04`	Container image used by the workload script
`LLM_API_KEY_FILE`	unset	File containing the log-analysis API key
`JUDGE_API_KEY_FILE`	unset	File containing the judge API key
`NVRX_LLM_MODEL`	`nvidia/nemotron-3-super-120b-a12b`	Model for log-analysis
`NVRX_LLM_BASE_URL`	`https://integrate.api.nvidia.com/v1`	Base URL for log-analysis
`JUDGE_MODEL`	`qwen/qwen3.5-397b-a17b`	Model for judge scoring
`JUDGE_BASE_URL`	`https://integrate.api.nvidia.com/v1`	Base URL for judge scoring
`FR_SEGMENT_SIZE`	`32`	Ranks per segment for coarse FR scoring
`SBATCH_SCRIPT`	`scripts/l4_gb200_reduced.sh`	Job script to submit
`POOL`	(default pool above)	Space-separated experiment triplets

Registered workloads (`scripts/workloads.conf`)

Name	Script	Base dir	Description
`llama4_scout`	`l4_gb200_reduced.sh`	`${HOME}/nvrx-attr-experiments`	Llama4-Scout (reduced layers) on GB200; minimum supported size is 2 nodes
`n3_super`	`n3_super_gb200_fi.sh`	`${HOME}/nvrx-attr-experiments`	Nemotron3-Super on GB200; minimum supported size is 8 nodes

Workload note:

llama4_scout requires at least 2 nodes.
n3_super requires at least 8 nodes. Its default registered pool contains only 8-node experiments.

# Run the full pool against the validated example workload
bash scripts/prepare_node_alloc.sh

# Run a custom subset against llama4_scout
POOL="GPU_SLEEP:1:5:2 SIGKILL:1:5:2" WORKLOAD=llama4_scout bash scripts/prepare_node_alloc.sh

Step 1 & 2 — Batched Submission + Wait (automated)

bash scripts/prepare_node_alloc.sh

The script loops: submit 2 jobs → poll squeue every 30 s until both finish → submit next 2. Progress is printed inline:

>>> Batch 1: experiments 1–2 of 34
  submitted: GPU_SLEEP rank=1  iter=5 nodes=2 -> job=1850
  submitted: GPU_SLEEP rank=0  iter=5 nodes=2 -> job=1851
  waiting for GPU_SLEEP:1:5:2 GPU_SLEEP:0:5:2 (1850,1851) ... 30s 60s done.
>>> Batch 2: experiments 3–4 of 34
  ...

A session directory and TSV tracking file are created at launch time:

${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/
  experiments.tsv                              ← tracking file (all job IDs + paths)
  n<N>_<FAULT>_r<R>_i<I>/                     ← one subdir per experiment
    logs/slurm/<JOB_ID>.launch.out
    logs/slurm/<JOB_ID>.*.1.main_workload.log  ← log-analysis input
    checkpoints/                               ← fr-analysis input (FR dumps)
    tensorboard/
  experiments_report.md                        ← generated by watch_and_analyze.sh

Tracking file columns: JOB_ID FAULT_TYPE RANK ITER NODES EXPERIMENT_DIR

Step 3 — Analyze All Experiments

Run the watcher/analyzer — it reads the tracking file and processes each experiment as its job state leaves RUNNING/PENDING (works whether jobs are still running or already done):

bash scripts/watch_and_analyze.sh \
    ${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/experiments.tsv

The watcher:

Reads each row from the tracking TSV
Calls nvrx_logsage.py --exclude_nvrx_logs and parses the text output to get restart_decision and attribution_text
Calls FR analysis as python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*" and passes the raw table output to the judge
Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)
Appends a scored row to <session>_report.md as a markdown table row
Repeats until all experiments are analyzed

To also run the sub-skills interactively for a single experiment:

/log-analysis --log-path "${EXPERIMENT_DIR}/logs/slurm/${JOB_ID}.*.1.main_workload.log"
/fr-analysis  --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*"

Step 4 — Score Each Experiment

Scoring is performed by scripts/score_attribution.py, an LLM judge that receives the ground truth, the filtered raw log, the logsage attribution output, and the FR analysis output, then returns structured JSON scores with a reasoning note.

Column	Values	Meaning
restart_correct	`true` / `false` / `N/A`	Restart decision matches expected for this fault type
rank_primary	`true` / `false` / `partial`	Injected rank is the primary root-cause in attribution
rank_any	`true` / `false`	Injected rank mentioned anywhere in attribution
fault_described	`true` / `false` / `partial`	Fault nature (hang/crash/signal/exception) correctly described
fr_rank_correct	`rank` / `node` / `segment` / `false` / `no_dumps`	FR analysis narrows correctly to the injected rank, exactly one `GPUS_PER_NODE` rank block containing that rank, the configured `FR_SEGMENT_SIZE` rank block containing the injected rank, or fails to narrow usefully
judge_notes	string	One-sentence summary of the main gap or confirmation

The judge is given:

Ground truth: fault_type, rank, iter, nodes
Expected restart decision + rationale (derived from score_attribution.py:_RESTART_TABLE)
Filtered raw log (last 400 lines, same exclude_nvrx_logs filtering as logsage)
Raw logsage stdout (5-field text format)
Raw FR analysis table output from fr_attribution.py --fr-path ... -p "_dump_*"
GPUS_PER_NODE and FR_SEGMENT_SIZE to map the injected rank to exact node-sized and segment-sized scopes for FR scoring

Default judge model: qwen/qwen3.5-397b-a17b. Override with --model in score_attribution.py. Default segment size for FR scope scoring: 32 ranks. Override with FR_SEGMENT_SIZE.

Step 5 — Aggregate Results

The canonical output of the loop is the markdown table in <session>_report.md. When summarizing results for users, prefer linking to that file and reproducing the same table shape rather than flattening the results into plain prose.

The report markdown table from watch_and_analyze.sh gives a matrix view. Look for patterns across rows:

| FAULT_TYPE | NODES | RANK | restart_correct | rank_primary | rank_any | fault_described | fr_rank_correct | judge_notes |
|------------|-------|------|-----------------|--------------|----------|-----------------|-----------------|-------------|
| GPU_SLEEP  |   2   |  0   |      true       |    false     |   true   |      true       |      true       | rank-0 identified only in secondary issues |
| GPU_SLEEP  |   2   |  1   |      true       |     true     |   true   |      true       |      true       | correct on all dimensions |
| GPU_ERROR  |   2   |  1   |      false      |    false     |  false   |     partial     |      true       | LLM issued RESTART; rank not mentioned |
| SIGKILL    |   2   |  0   |      true       |    false     |  false   |     false       |      true       | attribution describes timeout not kill signal |

Common failure mode patterns and their meaning:

Pattern	Interpretation
`rank_primary=false`, `rank_any=true`	Rank detected but treated as collateral; logsage putting it in secondary issues
`rank_any=false` for rank-0	Rank-0 hang silences watchdog on other ranks; logsage lacks rank-0 signal
`fault_described=partial` for crash types	Crash keywords present but fault type not specifically named
`restart_correct=false` for GPU_ERROR	LLM conflating hardware error with recoverable hang
`fr_rank_correct=no_dumps`	NCCL watchdog did not fire before job ended — adjust `TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC`
`fr_rank_correct=node`	FR isolated exactly one `GPUS_PER_NODE` rank block containing the injected rank, but not the exact rank
`fr_rank_correct=segment`	FR isolated the configured `FR_SEGMENT_SIZE` rank block containing the injected rank, but not the exact node/rank

Step 6 — Identify and Apply Improvements

FR analysis

Deterministic graph algorithm — do not modify automatically. Note misidentifications and escalate to the team.

Log analysis (safe to modify)

Observation	Target location	Suggested fix
Wrong restart for hang	`nvrx_logsage.py` fast-path	Strengthen NCCL timeout → `RESTART IMMEDIATE` mapping
Missing rank in attr text	`nvrx_logsage.py` prompt	Extract rank from NCCL watchdog lines; add regex
Crash misclassified as hang	`nvrx_logsage.py`	Add SIGKILL/SEGFAULT/GPU_ERROR keyword patterns
`ERRORS NOT FOUND` when errors exist	`return_application_errors` config	Loosen error extraction filter
rank-0 not detected	prompt or fast-path	Add explicit rank-0 hang heuristic (other ranks silent)
attr off by many iters	prompt context	Increase weight of iteration-stamped log lines
LLM wrong on GPU_ERROR	prompt	Distinguish `cudaError` → crash from NCCL timeout → hang

Editable file: attribution/log_analyzer/nvrx_logsage.py

After each patch, re-run the same pool subset that previously failed:

POOL="GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2" bash scripts/prepare_node_alloc.sh

Step 7 — Loop

Increment experiment counter. Suggested sweep order across code-change iterations:

Iteration 1: full default pool (34 experiments)
Iteration 2: targeted re-run of all failing cells from iteration 1
Iteration 3: expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps
Iteration 4: add SEGFAULT and LOCK_GIL 4-node/8-node coverage

Stop condition: all cells pass all four scoring dimensions for two consecutive code-change iterations.

Adapting A SLURM Script For The Feedback Loop

The feedback loop is not tied to l4_gb200_reduced.sh, but your sbatch script must match a small contract so the loop can submit, analyze, and score each run.

Required changes for a custom workload script:

Accept these exported variables from prepare_node_alloc.sh: FAULT_TYPE, FAULT_RANK, FAULT_AT_ITER, EXPERIMENT_DIR, BASE_EXPERIMENTS_DIR, and GPUS_PER_NODE.
Write the main training log to: ${EXPERIMENT_DIR}/logs/slurm/${SLURM_JOB_ID}.*.1.main_workload.log so watch_and_analyze.sh can find it.
Write NCCL flight-recorder dumps under ${EXPERIMENT_DIR}/checkpoints/.
Emit a fault-injection marker when the fault is injected. watch_and_analyze.sh uses this to decide whether the run reached the injection point.
Preserve the per-experiment directory layout: logs/slurm/, checkpoints/, and tensorboard/.

This has only been validated with Megatron-LM because the current run-valid check and fault markers depend on Megatron's debug_fault_injection.py behavior. If you adapt the loop to another framework, update both the sbatch script and watch_and_analyze.sh.

Appendix A: SBATCH_SCRIPT fault parameters

The example SBATCH_SCRIPT reads these env vars from prepare_node_alloc.sh via --export:

Variable	Default	Description
`FAULT_AT_ITER`	`5`	Training iteration at which to inject
`FAULT_DELAY`	`15`	Delay in seconds before fault injection after the iteration anchor
`FAULT_RANK`	`1`	Global rank to inject `[0, total_ranks)`
`FAULT_TYPE`	`GPU_SLEEP`	Megatron fault type enum name
`GPUS_PER_NODE`	`4`	GPUs per node (used to compute `TOTAL_TASKS`)
`EXPERIMENT_DIR`	`${BASE_EXPERIMENTS_DIR}/fault_injection/n${SLURM_NNODES}_${FAULT_TYPE}_r${FAULT_RANK}_i${FAULT_AT_ITER}`	Per-experiment output root
`BASE_EXPERIMENTS_DIR`	`${HOME}/nvrx-attr-experiments`	Shared root (datacache, triton/inductor caches)

Valid FAULT_TYPE values: GPU_ERROR, GPU_SLEEP, WORKLOAD_EXC, ASYNC_EXC, SIGNAL_EXC, OS_ABORT, LOCK_GIL, SEGFAULT, SIGINT, SIGKILL, SIGTERM, SIGSTOP

Appendix B: Single-experiment manual run

# Manual runs land under fault_injection/manual/ by default (no session dir needed)
EXPERIMENT_DIR=${HOME}/nvrx-attr-experiments/fault_injection/manual/n2_GPU_SLEEP_r1_i5
mkdir -p ${EXPERIMENT_DIR}/logs/slurm ${EXPERIMENT_DIR}/checkpoints ${EXPERIMENT_DIR}/tensorboard

sbatch \
    --nodes=2 \
    --output=${EXPERIMENT_DIR}/logs/slurm/%j.launch.out \
    --error=${EXPERIMENT_DIR}/logs/slurm/%j.launch.err \
    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=${EXPERIMENT_DIR} \
    scripts/l4_gb200_reduced.sh

Optional site-specific cleanup:

export CONTAINER_CLEANUP_CMD='
ENROOT_DIR="/var/lib/enroot/data/$(id -u)"
rm -rf "${ENROOT_DIR:?}"/* 2>/dev/null || true
echo "$(hostname): / $(df -h / | tail -1 | awk "{print \$3\" used, \"\$4\" avail\"}")"
'