update-golden-values

Update golden values + relative-diff summary

End-to-end workflow for refreshing golden values from a GitHub Actions workflow run, scoring the update with a per-metric average normalized relative difference, and writing a PR-ready summary.

The skill orchestrates two scripts that already live in the repo:

• tests/test_utils/python_scripts/download_golden_values.py — pulls artifacts from a workflow run and overwrites tests/functional_tests/test_cases/**/golden_values_*.json.
• tests/test_utils/python_scripts/compare_golden_values_kl.py — diffs the working-tree goldens against git HEAD and reports per-metric avg_rel_diff = mean((old − new) / old). (Filename keeps the legacy _kl suffix; the script no longer computes KL divergence.)

Inputs to gather from the user

1. GitHub Actions workflow run ID (e.g. 25341543542). It's the numeric ID in the run URL.

2. Source: should be github for this workflow. (gitlab is supported by the download script but uses a different env path.)

3. Scope — accept one of:

   • only-failing → run with --only-failing (download from failing/cancelled jobs only). Use this for "fix the broken tests" workflows.
   • all → run without --only-failing (download from every job that produced golden values). Use this when the user wants a full refresh.

   If the user doesn't specify, ask. Don't silently default.

Workflow

- [ ] Step 1: Set up env (token + venv with deps)
- [ ] Step 2: Reset prior golden-value edits
- [ ] Step 3: Download goldens (scope = only-failing | all)
- [ ] Step 4: Run relative-diff comparison + capture CSV
- [ ] Step 5: Produce summary blurb

Step 1 — Environment

The download script needs GITHUB_TOKEN. If the user has the gh CLI authenticated, derive it; do NOT export the token into a long-lived shell or commit it.

# token (one-shot, scoped to the command)
export GITHUB_TOKEN="$(gh auth token)"

# python deps (the script imports click, gitlab, requests)
python3 -m venv /tmp/gv_venv
/tmp/gv_venv/bin/pip install --quiet click python-gitlab requests

Reuse /tmp/gv_venv if it already exists. The comparison script only depends on click (also in the venv).

Step 2 — Reset prior edits (only if user re-runs)

If the working tree already has prior golden-value modifications you want to discard before re-downloading:

git checkout -- tests/functional_tests/test_cases/
git ls-files --others --exclude-standard tests/functional_tests/test_cases/ \
  | while IFS= read -r f; do rm -f "$f"; done

Skip this step when the user explicitly wants to layer a new download on top of an in-progress branch.

Step 3 — Download

Build the command from the user-provided scope:

# scope = only-failing (default for "fix broken tests")
/tmp/gv_venv/bin/python tests/test_utils/python_scripts/download_golden_values.py \
  --source github --pipeline-id <WORKFLOW_RUN_ID> --only-failing

# scope = all (full refresh; omit the flag)
/tmp/gv_venv/bin/python tests/test_utils/python_scripts/download_golden_values.py \
  --source github --pipeline-id <WORKFLOW_RUN_ID>

When --only-failing is set, the GitHub path filters at _fetch_and_filter_artifacts on matched_job["conclusion"] == "success", so only failing/cancelled jobs contribute artifacts. Without the flag, every job's golden-value artifact is pulled.

Capture the final two log lines for the summary; they look like:

INFO:__main__:Total tests with golden values: <N>
INFO:__main__:Total golden values found: <M>

Step 4 — Relative-diff comparison

/tmp/gv_venv/bin/python tests/test_utils/python_scripts/compare_golden_values_kl.py \
  --top 20 --csv /tmp/reldiff_summary.csv

The CSV holds one row per (file, metric) with four columns:

file, metric, n_steps, avg_rel_diff

• n_steps — count of shared steps that contributed (steps where |old| < 1e-12 are skipped to avoid div-by-zero; NaN/inf are dropped).
• avg_rel_diff — mean((old − new) / old). Signed: positive = the new run is smaller than the old run at the typical step (e.g. loss decreased), negative = larger.

Then derive aggregates from the CSV (do this in Python; do not paste raw CSV into the summary):

import csv, collections
rows = list(csv.DictReader(open('/tmp/reldiff_summary.csv')))
for r in rows:
    r['n_steps']      = int(r['n_steps'])
    r['avg_rel_diff'] = float(r['avg_rel_diff'])
    r['abs']          = abs(r['avg_rel_diff'])

by_metric = collections.defaultdict(list)
for r in rows:
    by_metric[r['metric']].append(r['abs'])

# headline numbers per metric (using |avg_rel_diff|)
for m, vs in sorted(by_metric.items()):
    vs.sort()
    print(m, len(vs), 'median', vs[len(vs)//2], 'max', vs[-1])

# bucket counts across all rows, on |avg_rel_diff|
buckets = [('==0',      lambda x: x == 0),
           ('(0,1e-6)', lambda x: 0 < x < 1e-6),
           ('[1e-6,1e-4)', lambda x: 1e-6 <= x < 1e-4),
           ('[1e-4,1e-3)', lambda x: 1e-4 <= x < 1e-3),
           ('[1e-3,1e-2)', lambda x: 1e-3 <= x < 1e-2),
           ('[1e-2,1e-1)', lambda x: 1e-2 <= x < 1e-1),
           ('>=1e-1',   lambda x: x >= 1e-1)]
abs_all = [r['abs'] for r in rows]
for label, pred in buckets:
    print(label, sum(1 for v in abs_all if pred(v)))

Step 5 — Summary blurb

Use this template verbatim, filling in <…> from steps 3–4. Drop sections that don't apply to the run.

Pick the wording for the first line based on the scope used:

• only-failing → "Refresh of golden values for failing functional tests from GitHub workflow run …"
• all → "Full refresh of golden values from GitHub workflow run …"

Match the download_golden_values.py command in the bullet list to the scope used (with or without --only-failing).

### Summary

<scope-appropriate sentence> from GitHub workflow run `<WORKFLOW_RUN_ID>`.

**Golden value updates**

- Re-ran `tests/test_utils/python_scripts/download_golden_values.py --source github --pipeline-id <WORKFLOW_RUN_ID> <--only-failing if scope=only-failing>`.
- Updated **<N> golden-value files** under `tests/functional_tests/test_cases/`.

### Relative-difference summary

Comparison covers <FILES_WITH_BASELINE> files × <NUM_METRICS> metrics = **<TOTAL_ROWS> `(file, metric)` pairs**. Per row: `avg_rel_diff = mean((old − new) / old)` over shared steps.

**Per-metric headline numbers** (over `|avg_rel_diff|`)

| metric                    |   n | median \|avg_rel_diff\| | max \|avg_rel_diff\| |
| ------------------------- | --: | -----------------------: | -------------------: |
| `lm loss`                 | <…> |                    <…>   |                <…>   |
| `num-zeros`               | <…> |                    <…>   |                <…>   |
| `iteration-time`          | <…> |                    <…>   |                <…>   |
| `mem-allocated-bytes`     | <…> |                    <…>   |                <…>   |
| `mem-max-allocated-bytes` | <…> |                    <…>   |                <…>   |

**Distribution of `|avg_rel_diff|` across all <TOTAL_ROWS> rows**

| \|avg_rel_diff\| bucket | count |
| ----------------------- | ----: |
| `== 0`                  |  <…>  |
| `(0, 1e-6)`             |  <…>  |
| `[1e-6, 1e-4)`          |  <…>  |
| `[1e-4, 1e-3)`          |  <…>  |
| `[1e-3, 1e-2)`          |  <…>  |
| `[1e-2, 1e-1)`          |  <…>  |
| `>= 1e-1`               |  <…>  |

**Interpretation** (apply only the bullets that match the data)

- `lm loss` max `|avg_rel_diff|` <X> / median <Y> — loss trajectories match old goldens to numerical noise (sub-1e-4 is within run-to-run variance).
- `mem-*` metrics typically sit at `== 0` or `(0, 1e-6)`; flag any row that lands above `[1e-4, 1e-3)`.
- `iteration-time` movement is dominated by warmup/scheduler noise; signed avg near zero means the run was simply jitterier, not slower or faster on average.
- `num-zeros` shifts cluster on `<list of test patterns>`; within historical run-to-run variance.

Reading the columns

When sorting / filtering, the script ranks by |avg_rel_diff|. Keep the sign in the printed table so reviewers can see direction.

Triage rules of thumb:

• lm loss / num-zeros rows with |avg_rel_diff| ≲ 1e-4 are run-to-run noise.
• iteration-time divergences are usually warmup/scheduler noise; a small signed mean near zero says the run was jitterier, not systematically faster or slower.
• Focus reviewer attention on lm loss and num-zeros rows with |avg_rel_diff| ≥ ~1e-3.

Notes & gotchas

• The download script's _fetch_and_filter_artifacts honors --only-failing only on the GitHub path. The Gitlab path applies it per-job inside download_from_gitlab.
• A brand-new golden file (no git HEAD baseline) is silently skipped by the comparison script with a warning. Subtract these from the file count when reporting "files with baseline".
• Steps where |old| is below 1e-12 are excluded from the average — division blows up there (think num-zeros step 0 on a dense model, or mem-* before allocation). If every shared step is excluded for a metric, that (file, metric) row is omitted entirely.
• Some artifacts have a literal string "nan" in step 1 of iteration-time; the comparison script filters those out, so other steps for that metric still contribute. Don't flag iteration-time as a correctness problem unless something else also moved.
• The script's filename is compare_golden_values_kl.py for legacy reasons; it no longer computes KL divergence. The function and CSV column names reflect what it actually does (avg_rel_diff).
• Never commit GITHUB_TOKEN, RO_API_TOKEN, or any value derived from gh auth token. If the user wants you to commit, only stage golden-value files and the optional CSV — not the env or the venv.