agent-host-e2e-tests

Agent host end-to-end tests

These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.

Before doing anything, read the architecture + troubleshooting reference: src/vs/platform/agentHost/test/node/e2e/README.md

It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.

Non-negotiable rules

1. Replay is default and strict. No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
2. A model-backed fixture's filename is derived from the test title (${provider}-${slug}.yaml). Renaming such a test orphans its fixture — re-record after any rename. Tests explicitly registered with hostOnlyTest(...) share captures/empty.yaml.
3. Recording needs a real token (GITHUB_TOKEN or gh auth token) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs.
4. Never hand-write or hand-edit fixture contents (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
5. Gate, don't fight. If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
6. Track every disabled variant. Keep e2e/KNOWN_ISSUES.md current with the test title, scope, expected and observed behavior, and a focused reproduction command. For suspected product bugs, begin with a self-contained explanation in complete sentences of what the user is trying to do, what fails, and the likely user impact; define feature-specific terms instead of relying on test names or implementation details. Record symptoms, not speculative root causes.
7. Always verify new end-to-end tests in Azure DevOps. Every new Agent Host E2E or real-provider integration test must pass a focused build from VS Code pipeline definition 111 before merge. Local runs and GitHub pull-request CI are useful signals, but neither substitutes for the Azure build because its packaged Electron path and runner environments differ.

Workflow A — Add a cross-provider test

1. Add a test to the closest module under e2e/suites/, or create and register a focused suite module when the behavior is distinct. Use hostOnlyTest(context, ...) when crossing the model boundary would be a bug; otherwise use test(...). Drive turns with dispatchTurn(...) + context.client.waitForNotification(...); assert on AHP notifications, never on wall-clock timing.
2. Keep the prompt minimal and deterministic (fewer model turns → smaller, more robust fixtures).
3. Record fixtures for every enabled provider (Workflow B). Host-only tests need no per-test recording: the shared empty fixture remains strict and fails on any model request.
4. Review the diff (Workflow B step 3), then run the test in plain replay mode to confirm it's green, then commit the test + fixtures together.
5. Run the full deterministic suite and coverage workflow described in the E2E README.
6. Open or update a draft PR, then complete the cross-platform Azure validation in Workflow D before considering the tests ready to merge.

Provider-specific assertions go in that provider's *.integrationTest.ts after the defineAgentHostE2ETests(config) call.

Workflow B — Record / re-record fixtures

Re-record when you add a test, or when a bundled SDK/CLI bump changes its wire behavior (new endpoint, different turn count, changed tool schema).

1. Ensure a token is available: gh auth token (or export GITHUB_TOKEN).
2. Record per provider:

   AGENT_HOST_REPLAY_RECORD=1 ./scripts/test-integration.sh --run \
     src/vs/platform/agentHost/test/node/e2e/providers/claudeAgentHostE2E.integrationTest.ts
   

   Repeat for copilotAgentHostE2E / codexAgentHostE2E as needed.
3. Review git diff on the fixtures: no local usernames/absolute paths, no tokens, no unreleased model ids. If something leaked, the fix is to extend normalization/redaction in capiReplayProxy.ts (_normalize + the *_RE redactors) and re-record — not to edit the fixture.
4. Run plain replay (no env var) to confirm green, then commit.

If an SDK now hits a new ancillary/bootstrap endpoint (a probe, not a real model turn), add it to capiStubs.ts (served, not recorded) instead of recording it — see how /models/session is handled.

Workflow C — When a test can't replay deterministically

Real-time streaming, mid-turn aborts, and POSIX-specific local execution (shell tools, pwd, git worktrees) don't replay reliably. Gate them precisely so you keep coverage where it works:

• Record-only (no deterministic replay at all): (RECORD ? test : test.skip)('…') — see can abort a running turn.
• Subagent fixtures stale after an SDK bump: re-record them (AGENT_HOST_REPLAY_RECORD=1 …). Subagent flows are the most SDK-version-sensitive (parent + child share one /v1/messages sequence), but replay reliably once re-recorded, so no gating is needed.
• POSIX-only (fails on Windows): gate with !isWindows, or a targeted per-provider flag when only one provider diverges. See the worktree and subagent-reopen tests.
• Provider/OS-specific replay: add a targeted config gate that still permits recording and unaffected platforms. See the Codex shell-tool Linux gate.

Always add a comment explaining why the gate exists. Also add or update the corresponding entry in e2e/KNOWN_ISSUES.md. When the variant is enabled again, remove or update the entry in the same change.

Workflow D — Cross-platform Azure validation

New Agent Host E2E and real-provider integration tests are not ready to merge after local or GitHub pull-request CI alone. Push the branch, open or update a draft PR, then use the azure-pipelines skill to validate the real packaged Electron integration-test path.

1. Queue VS Code pipeline definition 111 with VSCODE_BUILD_TYPE=CI; enable Windows, Linux, and macOS x64 while disabling publishing, release, Web, ARM, Alpine, and Snap artifacts. The azure-pipelines skill contains the canonical command.
2. Monitor jobs as they finish. Inspect a failed platform's Electron integration-test task immediately rather than waiting for unrelated stages to complete.
3. Treat the Agent Host E2E result as accepted only when the Electron integration tests succeed on Windows, Linux, and macOS.
4. Rerun an apparently unrelated or pre-existing failure in isolation before attributing it to the PR.
5. After a platform-specific fix, rerun at least that platform. Rerun all three platforms when the fix can affect shared behavior, provider fixtures, process lifecycle, or cross-platform paths.
6. Cancel obsolete builds after pushing a replacement commit.

For additions involving timing, filesystem watching, process lifecycle, worktrees, reconnect/restart, or other known flake surfaces, require two clean executions of every new test on each supported platform before merge. A full three-platform build plus a targeted second build is sufficient when the second build runs the relevant tests on all affected platforms.

Verifying & troubleshooting

• Run a single provider in replay: ./scripts/test-integration.sh --run <path> (no env var).
• Filter to one test: add --grep "<test title fragment>".
• On a hang / timeout, read the runtime log first. For the Copilot provider, a failed test tails the most recent Copilot runtime (@github/copilot CLI) process-*.log into the test output ([agent-host-e2e] # … lines) — the SDK/CLI's own account of startup, auth, the model request, and the turn lifecycle. It runs at --log trace. A turn that never produced a model response, a panic, or an out-of-order/protocol error points at the SDK/CLI (re-record if a bump left the fixture stale; otherwise it's a real regression). Claude/Codex use their own runtimes and are not captured here. See the README's "A turn hangs or times out with no OS pattern".
• For any failure (cache miss, missing fixture, per-OS timeout, leaked PII, subagent staleness, accidental real-CAPI contact), go to the Troubleshooting section of the README — it maps each symptom to its cause and fix.