update-screenshots

Update Component Screenshots from CI

Screenshot images are not stored in the repository — they live in an external service (hediet-screenshots.azurewebsites.net), keyed by commit SHA. But a subset of fixtures is pinned by hash in test/componentFixtures/blocks-ci-screenshots.md, and that file is committed. When those hashes change, CI fails and you must update the file.

Two different outcomes, only one of which blocks

The Screenshots & Tests job in .github/workflows/component-fixtures.yml produces two independent results:

A fixture opts into the blocking gate with labels: { kind: 'screenshot', blocksCi: true }. Only those fixtures appear in blocks-ci-screenshots.md.

The failure looks like this:

##[error]blocks-ci screenshot hashes do not match committed file. See PR comment or job summary for the updated content.

Step 1: Get the expected hashes from CI

Never regenerate the hashes locally. They are hashes of the rendered PNG bytes, produced on ubuntu-latest. Rendering on macOS or Windows yields different bytes and therefore different hashes, so locally generated values will fail CI. Always copy the values from the CI job.

Three surfaces carry the same content — use whichever is handy:

• The PR comment titled "blocks-ci screenshots changed" (non-fork PRs only) — contains the full updated file plus a patch.
• The job summary, which gets the identical body and is the only surface fork PRs receive.
• The job log, whose final step prints a unified diff:

gh api repos/microsoft/vscode/actions/jobs/<JOB_ID>/logs > "$TMPDIR/ci-job-log.txt"
grep -n '##\[error\]' "$TMPDIR/ci-job-log.txt"

Find the failed job id with:

gh pr checks <PR> --json name,link,bucket --jq '.[] | select(.name == "Screenshots & Tests")'

Step 2: Verify the change is intentional before accepting it

This gate exists to catch unintended layout regressions, so accepting new hashes without looking at the images defeats its purpose. The images are publicly fetchable by hash, so pull both the old (committed) and new (from CI) versions and compare:

curl -sL -o old.png "https://hediet-screenshots.azurewebsites.net/images/<OLD_HASH>"
curl -sL -o new.png "https://hediet-screenshots.azurewebsites.net/images/<NEW_HASH>"

Then view them, and localize the change rather than eyeballing full screenshots — the delta is often only a pixel or two:

python3 -c "
from PIL import Image, ImageChops
a = Image.open('old.png').convert('RGB'); b = Image.open('new.png').convert('RGB')
print('diff bbox:', ImageChops.difference(a, b).getbbox())
"

Confirm the delta matches what the PR intends. If the fixture is unrelated to the change, or the shift is larger than expected, treat it as a regression and fix the code instead of the hashes.

Step 3: Apply and commit

Edit only the changed lines in test/componentFixtures/blocks-ci-screenshots.md, replacing the old hash in the image URL with the new one:

#### editor/inlineChatZoneWidget/InlineChatZoneWidget/Dark
![screenshot](https://hediet-screenshots.azurewebsites.net/images/<NEW_HASH>)

The file is generated by build/lib/screenshotBlocksCi.ts and compared byte-for-byte, so keep the <!-- auto-generated by CI — do not edit manually --> header, the #### <fixtureId> / image-link pairing, the blank line between entries, and the fixtureId sort order intact. Verify your edit is the exact inverse of the diff CI reported:

git diff test/componentFixtures/blocks-ci-screenshots.md

Then commit and push. The check re-runs and should pass; hashes on main become the new baseline after merge.

Investigating further

Raw captured images and the manifest for a run are uploaded as an artifact:

gh run download <RUN_ID> --name screenshots --dir .tmp/screenshots

manifest.json maps each fixtureId to its imageHash and any render errors.

Related failures from the same job

The check also fails if a fixture failed to render (Fail if fixtures had errors) or if the Playwright fixture tests failed. Those are genuine bugs — updating hashes will not help. Look for ::error::<fixtureId>: in the log, and download the playwright-test-results artifact for test failures.