log-analysis

작성자: nvidia

SLURM 작업 로그 파일을 분석하여 NVRxLogAnalyzer를 사용해 실패 원인을 파악하고 재시작 결정을 내립니다. SLURM 학습 작업 로그가 있고…

npx skills add https://github.com/nvidia/nvidia-resiliency-ext --skill log-analysis

ZIP 다운로드 GitHub

338

Skill: log_analysis

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

What it does

Reads the log file (UTF-8, falls back to latin-1).
Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
Optionally runs L1 model enrichment to explain the primary failure and related evidence.
Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

CLI

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \
    [--config examples/attribution/restart_agent.json] \
    [--llm-model MODEL] \
    [--llm-base-url URL] \
    [--llm-api-key-file PATH] \
    [--disable-l1] \
    [--summary]

Flag	Default	Description
`log_path`	required	Path to the job log file
`--job-id`	unset	Optional job id for request metadata and retry history
`--cycle-id`	unset	Optional cycle id for retry history
`--config`	unset	Versioned restart-agent config for configured model routes and retry policy
`--llm-model`	environment/default config	L1 enrichment model when `--config` is not used
`--llm-base-url`	environment/default config	OpenAI-compatible base URL when `--config` is not used
`--llm-api-key-file`	environment/default config	File containing the L1 API key
`--disable-l1`	off	Run deterministic-only analysis
`--summary`	off	Emit a compact human-readable summary to stderr in addition to JSON stdout

Programmatic API

from nvidia_resiliency_ext.attribution.restart_agent.l1 import LlmConfig, LlmEvidenceExtractor
from nvidia_resiliency_ext.attribution.restart_agent.models import RestartAgentRequest
from nvidia_resiliency_ext.attribution.restart_agent.pipeline import RestartAgent

agent = RestartAgent(evidence_extractor=LlmEvidenceExtractor(LlmConfig.from_env()))
run = agent.run(RestartAgentRequest(log_path="/path/to/job.log", job_id="12345"))
result = run.result.to_payload()

Output

The CLI prints one JSON object to stdout. Important fields include:

Field	Meaning
`decision`	`RESTART` or `STOP`
`decision_basis`	Policy reason for the decision
`retry_policy`	Retry budget and selected rule details
`primary_failure`	Primary failure evidence, when identified
`secondary_failures`	Related or cascading failure evidence
`evidence_coverage`	Which evidence sources were checked/found
`justification`	Human-readable explanation for the final decision

Prerequisites

LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
Use --disable-l1 when you want deterministic-only analysis without model access