test-release-canary

Test Release Canary

The Release Canary (.github/workflows/release-canary.yml) smoke-tests the artifacts a Release Dev run just published. It is the last automated checkpoint before tagging a public release: if the canary is red, the published dev artifacts do not install on a stock environment.

What the canary verifies

All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject OPENSHELL_TELEMETRY_ENABLED=false through the service environment, and the Kubernetes job installs with server.telemetryEnabled=false, so smoke traffic does not contribute to product usage metrics.

The workflow sets OPENSHELL_VERSION=dev for every install.sh job. Positive jobs consume the rolling dev release produced by the triggering workflow. The system-Docker Snap lane tracks latest/edge; the missing-Docker lane exits before installing a snap. The Debian and Kubernetes CLI lanes remove snapd so they continue to exercise the dev Debian package. Kubernetes pins the matching 0.0.0-dev chart and :dev images.

RPM package installation also has tmachine conformance coverage in Branch E2E Checks (with test:e2e), Release Dev, and Release Tag. Those lanes use the rpm installer on fedora-podman-rootful and fedora-podman-rootless with candidate CLI and gateway RPMs and matching runtime images. Branch RPM package builds run on every approved branch run, independently of optional E2E labels.

The host-package jobs exercise fresh installs, not upgrades from a persisted schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with the release-tooling and package lifecycle tests before relying on the canary.

The canary does not install or import @nvidia/openshell-sdk. TypeScript SDK validation lives in the TypeScript SDK branch check, including a publish dry-run. The tagged release workflow publishes the package to GitHub Packages; verify that job directly when diagnosing SDK publication failures.

Trigger paths

The workflow has two triggers:

on:
  workflow_dispatch:
  workflow_run:
    workflows: ["Release Dev"]
    types: [completed]

• Automatic. Every successful Release Dev run (on main or a manual dispatch of Release Dev) fires the canary. Each job gates on github.event.workflow_run.conclusion == 'success' so a failed Release Dev does not run the canary.
• Manual. workflow_dispatch lets you run the canary on demand against any branch's workflow definition.

When dispatched manually, github.event.workflow_run.head_sha is empty and the workflow falls back to github.sha (the branch tip) for the install.sh URL.

Manual dispatch

Run the canary as-is on the current branch:

gh workflow run release-canary.yml --ref "$(git branch --show-current)"

Watch the run that starts:

sleep 5  # let GitHub register the dispatch
gh run list --workflow release-canary.yml --limit 1
gh run watch "$(gh run list --workflow release-canary.yml --limit 1 --json databaseId --jq '.[0].databaseId')"

View only failed jobs after completion:

gh run view <run-id> --log-failed

Iterating on the canary itself

When you change release-canary.yml on a branch, a manual dispatch on that branch tests your branch's workflow logic against main's published dev artifacts (0.0.0-dev chart, :dev images, and the dev GitHub release). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.

Note install.sh is pulled from raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh, so changes to install.sh on your branch are exercised even though packages come from the latest public dev release and the snap comes from latest/edge.

Testing artifacts from a specific SHA

Release Dev publishes two chart versions for every dev build (see .github/actions/release-helm-oci/action.yml:89-102):

• oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev — floating, overwritten on every main push.
• oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha> — immutable, appVersion set to the same SHA so it pulls the matching gateway, sandbox, and supervisor images.

To smoke-test the chart for a specific dev build, dispatch Release Dev on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose chart_version / image_tag inputs.

Local kind reproduction

The kubernetes job can be reproduced on any machine with Docker and mise install-provided kubectl + helm:

kind create cluster --name release-canary-local

bash e2e/support/install-agent-sandbox.sh

helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
  --version 0.0.0-dev \
  --namespace openshell --create-namespace \
  --set server.disableTls=true \
  --set server.telemetryEnabled=false \
  --wait --timeout 5m

kubectl wait --namespace openshell \
  --for=condition=Ready pod \
  --selector="app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell" \
  --timeout=300s

kubectl port-forward --namespace openshell svc/openshell 8080:8080 &
openshell gateway add http://127.0.0.1:8080 --local --name kind
openshell status

Keep pkiInitJob.enabled=true (the chart default), even when server.disableTls=true. The hook also generates the sandbox JWT signing secret that the gateway pod always mounts.

Swap 0.0.0-dev for 0.0.0-dev.<sha> to pin to a specific dev build. Tear down with kind delete cluster --name release-canary-local.

Loopback registration auto-derives the gateway name to openshell if --name is omitted, which collides with the install.sh-installed local gateway — always pass --name kind (or another distinct name) when registering in addition to a local install.

Diagnosing failures

The kubernetes job's diagnostics step (only runs if: failure()) emits, in order: helm status, rendered manifest, kubectl get all, pod descriptions, pod logs (200 lines per container), port-forward log, gateway list, CLI version. Read it top-to-bottom — most failures fall out by the manifest or pod logs.

Related

• helm-dev-environment skill — local k3d-based dev environment (more featureful than the canary's kind cluster, but uses Skaffold-built local images, not published artifacts).
• watch-github-actions skill — generic gh run workflow monitoring.
• debug-openshell-cluster skill — runtime gateway/sandbox diagnostics that pair with the kind job's diagnostics dump.