fr-analysis

Analyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a…

Skill: fr_analysis

Analyze PyTorch NCCL flight-recorder (FR) dumps to identify the collective operation hang and isolate the ranks responsible, using CollectiveAnalyzer.

Script: scripts/fr_attribution.py → attribution/trace_analyzer/fr_attribution.py

What it does

Loads all FR dump files matching a glob pattern under --fr-path.
Parses each dump into Collective records (op type, ranks, process group, timing, state).
Groups collectives by process group and sequence ID across ranks to detect mismatches.
Identifies the wavefront — the process group boundary where collectives diverge — and returns the missing ranks at that boundary as the root-cause suspects.
Optionally runs an LLM pass (--llm-analyze) over the structured findings for a human-readable summary.

CLI

python scripts/fr_attribution.py \
    --fr-path /path/to/fr_dumps/ \
    [-p "_dump_*"] \
    [--verbose] \
    [--health-check] \
    [--llm-analyze] \
    [--model MODEL] \
    [--debug]

Flag	Default	Description
`--fr-path`	required	Path to a directory (or single file) containing FR dump files
`--pattern`, `-p`	`_dump_*`	Glob pattern for dump files within `--fr-path`
`--verbose`, `-v`	off	Print detailed per-rank collective tables
`--health-check`, `-c`	off	Include node health check results in output
`--llm-analyze`, `-l`	off	Pass structured findings to the LLM for a narrative summary
`--model`, `-m`	`nvidia/nemotron-3-super-120b-a12b`	LLM model (only used with `--llm-analyze`)
`--debug`	off	Convert binary trace files to JSON for inspection

Programmatic API

from nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution import CollectiveAnalyzer

analyzer = CollectiveAnalyzer({
    "fr_path": "/path/to/fr_dumps/",
    "pattern": "_dump_*",
    "verbose": False,
    "health_check": False,
    "llm_analyze": False,
    "model": "nvidia/nemotron-3-super-120b-a12b",
})
results = analyzer.run_sync({
    "fr_path": "/path/to/fr_dumps/",
})
# results: tuple[FRAnalysisResult | str, AttributionState]

Output

Returns (result, AttributionState) where result is the FR analysis table and describes:

The selected wavefront/front process group
Missing ranks at that process group (root-cause suspects)
Per-rank collective status tables (when --verbose)
Node health summary (when --health-check)
LLM narrative (when --llm-analyze)

AttributionState.STOP indicates the hang is unrecoverable; CONTINUE indicates the job may be restartable after isolating the identified ranks.

Dump file formats

Format	Notes
`_dump_*` files	PyTorch FR dump prefix pattern used by the feedback loop
Binary pickle / JSON payloads	Detected automatically; use `--debug` to convert binary traces to JSON

FR dumps are typically written to the directory specified by TORCH_NCCL_DEBUG_INFO_TEMP_FILE or triggered automatically on NCCL timeout.

Prerequisites

FR dump files produced by PyTorch NCCL (set TORCH_NCCL_TRACE_BUFFER_SIZE > 0)
LLM_API_KEY required only when using --llm-analyze
langchain-openai required only when using --llm-analyze
FR_DEBUG=1 env var enables verbose debug logging in the script