log-analysis

Skill: log_analysis

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

What it does

1. Reads the log file (UTF-8, falls back to latin-1).
2. Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
3. Optionally runs L1 model enrichment to explain the primary failure and related evidence.
4. Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

CLI

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \
    [--config examples/attribution/restart_agent.json] \
    [--llm-model MODEL] \
    [--llm-base-url URL] \
    [--llm-api-key-file PATH] \
    [--disable-l1] \
    [--summary]

Programmatic API

from nvidia_resiliency_ext.attribution.restart_agent.l1 import LlmConfig, LlmEvidenceExtractor
from nvidia_resiliency_ext.attribution.restart_agent.models import RestartAgentRequest
from nvidia_resiliency_ext.attribution.restart_agent.pipeline import RestartAgent

agent = RestartAgent(evidence_extractor=LlmEvidenceExtractor(LlmConfig.from_env()))
run = agent.run(RestartAgentRequest(log_path="/path/to/job.log", job_id="12345"))
result = run.result.to_payload()

Output

The CLI prints one JSON object to stdout. Important fields include:

Prerequisites

• LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
• Use --disable-l1 when you want deterministic-only analysis without model access