eagle3-triage

EAGLE3 Pipeline Triage

Diagnose failures in the 4-step EAGLE3 offline pipeline. This skill walks through each step, identifies the failure point, and provides actionable fixes.

Pipeline Overview

Step	Script	Purpose	Common failure area
task_0	common/vllm/query.sh	Data synthesis via vLLM server	Server startup, model loading, OOM
task_1	common/eagle3/dump_offline_data_vllm.sh (or _hf.sh / .sh)	Dump hidden states	Backend selection, OOM, unsupported arch
task_2	common/eagle3/train_eagle.sh	Train EAGLE3 draft head	Dependencies, training crash, export
task_3	common/specdec_bench/quick_check.sh	Benchmark acceptance rate	Engine startup, draft model loading

Step 0 — Locate the experiment

Ask the user for one of:

• Experiment directory (e.g., the --job-dir passed to launch.py or slurm.py)
• The model name / YAML they ran

Find recent experiments under the job directory:

ls -td experiments/cicd/cicd_* | head -10
# or wherever --job-dir was pointed

Each experiment directory contains one subdirectory per task (task_0 through task_3), each with a log file whose name varies by launch mode (Slurm: sbatch_*.out, local Docker: *.log).

Step 1 — Fetch logs for the failed task

Match the log files generally and read the tail of each — errors appear at the end:

find experiments/<exp_id>/ -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do
  echo "=== $f ==="; tail -200 "$f"; echo
done

Look for the first task with a non-zero exit code or error message.

Step 2 — Diagnose by step

task_0 failures (Data Synthesis)

How it works: Launches a vLLM OpenAI-compatible server, polls /health until ready, then runs query.py to generate synthetic prompt/response pairs. Output goes to /scratchspace/data/.

Error pattern	Root cause	Fix
Server never becomes healthy (hangs at health check)	Model too large for allocated GPUs, or vLLM startup crash	Check BF16 weight size vs total allocated GPU memory; increase TP and/or nodes.
CUDA out of memory during model load	Insufficient GPU memory	Reduce --max-model-len or increase --tensor-parallel-size
trust_remote_code error	Model requires custom code but flag not set	Add --trust-remote-code before the -- separator in task_0 args
Vocab / tokenizer error	Missing tokenizer cache (e.g., GPT-OSS-20B needs TIKTOKEN_RS_CACHE_DIR)	Set TIKTOKEN_RS_CACHE_DIR to a pre-populated cache path in the environment
Architecture not supported	vLLM version doesn't support this model	Try a newer vLLM container (vllm/vllm-openai:latest)
CANCELLED ... DUE TO TIME LIMIT	Job wall-clock limit too short	Increase Slurm --time. Note: afterany deps let task_1 still start.
Empty /scratchspace/data/	query.py ran but produced no output	Check --data path exists and contains prompts. Check query.py logs.

task_1 failures (Hidden State Dump)

How it works: Loads the target model and runs a forward pass on each conversation, saving hidden states as .pt files in /scratchspace/offline_hidden_states/.

Three backends are available:

Backend	Script	When to use
vLLM	dump_offline_data_vllm.sh	Broad model coverage; uses vLLM's native hidden-state extractor
HF	dump_offline_data_hf.sh	VLMs, custom-code models, SWA attention; uses device_map="auto"
TRT-LLM	dump_offline_data.sh	Pure-text models with TRT-LLM support; needs --tp/--moe-ep args

Error pattern	Root cause	Fix
No such file or directory: dump_offline_data_vllm.sh	Wrong script path in YAML	Use the correct path under common/eagle3/
FileNotFoundError: /scratchspace/data	task_0 failed or produced no output	Re-run task_0 first, or point --input-data to existing data
CUDA out of memory	Model too large	Switch to _hf.sh (device_map="auto") or increase TP
RuntimeError / unsupported arch	Model not supported by TRT-LLM backend	Switch to dump_offline_data_hf.sh or dump_offline_data_vllm.sh
NCCL timeout / NCCL error	Multi-node communication failure	Retry. Reduce EP.
No .pt files in output dir	Script ran but extraction produced nothing	Check --max-seq-len and input data format
pyxis: child terminated with signal 15	SIGTERM — likely OOM	Increase TP or switch backends

task_2 failures (Training)

How it works: Installs requirements, runs launch_train.sh (Accelerate + FSDP) with the config from modelopt_recipes/general/speculative_decoding/eagle3.yaml, then exports via export_hf_checkpoint.py. Output: /scratchspace/eagle3/ and /scratchspace/export/.

Error pattern	Root cause	Fix
FileNotFoundError: /scratchspace/offline_hidden_states	task_1 failed or produced no output	Re-run task_1 first
CUDA out of memory during training	Batch size too large	Reduce training.train_bs or training.training_seq_len
KeyError / AttributeError in model loading	Model architecture not recognized by EAGLE3	Model may need code changes in modelopt for this architecture
Loss is NaN or diverges	LR too high or data quality issue	Reduce training.lr. Check hidden state data.
export_hf_checkpoint.py fails	Training produced incomplete checkpoint	Check /scratchspace/eagle3/ for model.safetensors

task_3 failures (Benchmark)

How it works: Launches vLLM with the target + draft model, runs acceptance rate and throughput benchmarks. Output: JSON files.

Error pattern	Root cause	Fix
FileNotFoundError: /scratchspace/export	task_2 failed or export step failed	Re-run task_2. Check export output.
trust_remote_code error at benchmark	Model requires it but quick_check.sh doesn't forward the flag	Pass --trust-remote-code in task_3 args
Server fails with draft model	Draft model config incompatible with engine	Check eagle_config.json and engine version
AR below threshold / exit code 1	Draft model quality too low	More epochs, data, or hyperparameter tuning
CUDA out of memory	Target + draft exceeds GPU memory	Increase TP
vLLM EAGLE3 not supported	vLLM version too old	Use a newer vLLM container

Step 3 — Check for new-model-specific issues

If the user is adding support for a new model, also check:

1. Is the model a VLM? → Use dump_offline_data_hf.sh (text-only path, no vision encoder invoked)
2. Does the model use sliding window attention (SWA)? → TRT-LLM backend won't work; use HF or vLLM
3. Does the model need trust_remote_code? → Add to task_0 args AND task_3 args
4. Is the model MoE? → Check eagle_config.json intermediate_size matches model's moe_intermediate_size
5. Is the model architecture recognized by EAGLE3 training? → may need code changes in modelopt/torch/speculative/
6. Custom tokenizer? → May need additional environment vars (e.g., TIKTOKEN_RS_CACHE_DIR)

Step 4 — Suggest fix and next steps

After diagnosis, provide:

1. Root cause — one-line summary
2. Fix — specific config change, code edit, or command to run
3. How to re-run — skip earlier successful steps by pointing to existing scratchspace artifacts

To skip task_0 and task_1 and re-run from task_2:

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_1.skip=true \
    --yes

To run only task_1 standalone (using existing task_0 data):

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_2.skip=true \
    pipeline.task_3.skip=true \
    --yes

If the fix requires code changes in ModelOpt (e.g., supporting a new model architecture), note that a separate PR in the modelopt repo is needed.

Step 5 — Record the failure pattern

If you encounter a failure pattern not seen before, capture it in the team's internal triage tracker — the symptom, root cause, and fix — so the next engineer debugging the same issue benefits.