log-analysis

bởi nvidia

Phân tích tệp nhật ký công việc SLURM để xác định nguyên nhân gốc rễ của lỗi và đưa ra quyết định khởi động lại bằng NVRxLogAnalyzer. Sử dụng khi bạn có nhật ký công việc đào tạo SLURM và…

npx skills add https://github.com/nvidia/nvidia-resiliency-ext --skill log-analysis

Tải ZIP GitHub

338

Skill: log_analysis

Analyze a SLURM job log file for failure root-cause attribution and restart decisions using RestartAgent.

Command: python -m nvidia_resiliency_ext.attribution.restart_agent.cli

What it does

Reads the log file (UTF-8, falls back to latin-1).
Builds deterministic L0 evidence from training progress, failure lines, checkpoint state, and known restart-policy signals.
Optionally runs L1 model enrichment to explain the primary failure and related evidence.
Applies the restart-agent retry/recovery policy and returns a structured STOP or RESTART decision.

CLI

python -m nvidia_resiliency_ext.attribution.restart_agent.cli \
    /path/to/job.log \
    [--job-id JOB_ID] \
    [--cycle-id CYCLE_ID] \
    [--config examples/attribution/restart_agent.json] \
    [--llm-model MODEL] \
    [--llm-base-url URL] \
    [--llm-api-key-file PATH] \
    [--disable-l1] \
    [--summary]

Flag	Default	Description
`log_path`	required	Path to the job log file
`--job-id`	unset	Optional job id for request metadata and retry history
`--cycle-id`	unset	Optional cycle id for retry history
`--config`	unset	Versioned restart-agent config for configured model routes and retry policy
`--llm-model`	environment/default config	L1 enrichment model when `--config` is not used
`--llm-base-url`	environment/default config	OpenAI-compatible base URL when `--config` is not used
`--llm-api-key-file`	environment/default config	File containing the L1 API key
`--disable-l1`	off	Run deterministic-only analysis
`--summary`	off	Emit a compact human-readable summary to stderr in addition to JSON stdout

Programmatic API

from nvidia_resiliency_ext.attribution.restart_agent.l1 import LlmConfig, LlmEvidenceExtractor
from nvidia_resiliency_ext.attribution.restart_agent.models import RestartAgentRequest
from nvidia_resiliency_ext.attribution.restart_agent.pipeline import RestartAgent

agent = RestartAgent(evidence_extractor=LlmEvidenceExtractor(LlmConfig.from_env()))
run = agent.run(RestartAgentRequest(log_path="/path/to/job.log", job_id="12345"))
result = run.result.to_payload()

Output

The CLI prints one JSON object to stdout. Important fields include:

Field	Meaning
`decision`	`RESTART` or `STOP`
`decision_basis`	Policy reason for the decision
`retry_policy`	Retry budget and selected rule details
`primary_failure`	Primary failure evidence, when identified
`secondary_failures`	Related or cascading failure evidence
`evidence_coverage`	Which evidence sources were checked/found
`justification`	Human-readable explanation for the final decision

Prerequisites

LLM_API_KEY set (env var, LLM_API_KEY_FILE, or ~/.llm_api_key) for L1 enrichment
Use --disable-l1 when you want deterministic-only analysis without model access