helm-dev-environment
Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster…
npx skills add https://github.com/nvidia/openshell --skill helm-dev-environmentHelm Dev Environment
Set up, run, and tear down the local Kubernetes development environment for OpenShell.
The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).
Prerequisites
- Docker Desktop (macOS) or Docker Engine (Linux) running
mise installcompleted (providesk3d,kubectl,skaffold,helm)
Startup
1. Create the cluster
mise run helm:k3s:create
Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file.
Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION
in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox
releases) and preloads the default community sandbox image into k3d so the first sandbox
create does not wait on a large registry pull. Traefik is disabled at cluster creation time.
Multi-worktree support: the cluster name is derived from the last component of the
current git branch (e.g. branch kube-support/local-dev/tmutch → cluster
openshell-dev-tmutch). Each worktree therefore gets its own isolated cluster and its
own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name
or share one cluster across worktrees.
Port mappings created at cluster time (cannot be changed without recreating):
| Host port | Target | Used by |
|---|---|---|
8080 | Port 80 via k3d load balancer | Envoy Gateway LoadBalancer service (values-gateway.yaml) |
Override with env vars before running helm:k3s:create:
HELM_K3S_LB_HOST_PORT(default:8080)HELM_K3S_PRELOAD_SANDBOX_IMAGE(default:ghcr.io/nvidia/openshell-community/sandboxes/base:latest; set to an empty value to skip)
2. Deploy OpenShell
Iterative dev (rebuilds on file changes, recommended during active development):
mise run helm:skaffold:dev
One-shot deploy (build once and leave running):
mise run helm:skaffold:run
Supervisor sidecar topology (build once and leave running):
mise run helm:skaffold:run:sidecar
Supervisor sidecar topology with TLS/mTLS enabled (build once and leave running):
mise run helm:skaffold:run:sidecar-mtls
Both commands build the gateway and supervisor images and deploy the OpenShell Helm
chart. The sidecar profile renders an openshell-network-init init container for
nftables setup and an openshell-supervisor-network runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with SYS_PTRACE and
DAC_READ_SEARCH; relaxed mode can run it as the configured proxy UID. The
sidecar-mTLS profile reuses ci/values-sidecar.yaml and restores
server.disableTls=false inline for Skaffold. The pkiInitJob hook (a pre-install
Job that runs openshell-gateway generate-certs) generates mTLS secrets on first
install. Envoy Gateway opt-in; see the Optional Add-ons section below.
The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or kubectl port-forward.
HA test deploy (two gateway replicas + external PostgreSQL Secret): uncomment
#- ci/values-high-availability.yaml in deploy/helm/openshell/skaffold.yaml,
create the Secret named openshell-ha-pg with a uri key, then run
mise run helm:skaffold:run or mise run helm:skaffold:dev.
TLS behaviour
ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run
plaintext by default. To test sidecar topology with TLS enabled, use
mise run helm:skaffold:run:sidecar-mtls.
| Mode | server.disableTls | Gateway scheme |
|---|---|---|
| Skaffold dev (default) | true | http:// |
| TLS enabled | false (or omitted) | https:// |
Connecting via port-forward
Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so
the port-forward uses local port 8090 to avoid a collision:
KUBECONFIG=kubeconfig kubectl port-forward -n openshell svc/openshell 8090:8080
Plaintext (default Skaffold deploy):
openshell sandbox list --gateway-endpoint http://localhost:8090
With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:
mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
-o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
-o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
-o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key
The server cert SANs include localhost and 127.0.0.1, so hostname verification
passes over a port-forward without any extra flags:
openshell sandbox list --gateway-endpoint https://localhost:8090
Teardown
Remove the Helm releases (keep cluster)
mise run helm:skaffold:delete
For a sidecar-profile deployment:
mise run helm:skaffold:delete:sidecar
Delete the cluster entirely
mise run helm:k3s:delete
This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.
Optional Add-ons
Each add-on requires uncommenting the corresponding valuesFiles entry in
deploy/helm/openshell/skaffold.yaml before running helm:skaffold:dev or helm:skaffold:run.
Envoy Gateway (Gateway API / GRPCRoute)
Envoy Gateway is already installed by Skaffold (the envoy-gateway Helm release in
skaffold.yaml). To activate routing:
- Uncomment
#- values-gateway.yamlinskaffold.yaml - Redeploy:
mise run helm:skaffold:run - Apply the GatewayClass:
mise run helm:gateway:apply - Access:
http://127.0.0.1:8080
values-gateway.yaml creates a Gateway (listener on port 80, class eg) and a
GRPCRoute in the openshell namespace. Envoy Gateway provisions a LoadBalancer
service for the proxy; klipper-lb binds it to hostPort 80, reachable via the
8080:80 load balancer port mapping.
Keycloak OIDC
One-time setup — only needed once per cluster lifetime:
mise run keycloak:k8s:setup
This deploys Keycloak (quay.io/keycloak/keycloak:24.0) into the keycloak namespace,
imports the openshell realm from scripts/keycloak-realm.json, and prints a port-forward
command for acquiring tokens from the CLI.
Then activate OIDC in the OpenShell Helm chart:
- Uncomment
#- ci/values-keycloak.yamlinskaffold.yaml - Redeploy:
mise run helm:skaffold:run
To remove Keycloak:
mise run keycloak:k8s:teardown
SPIRE / SPIFFE Provider Token Grants
Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate SPIFFE JWT-SVIDs for dynamic provider token grants:
- Uncomment the
spire-crdsandspirereleases indeploy/helm/openshell/skaffold.yaml - Uncomment
#- ci/values-spire.yamlin the OpenShell release values files - Redeploy:
mise run helm:skaffold:run
ci/values-spire-stack.yaml configures the local SPIRE trust domain as
openshell.local and adds a ClusterSPIFFEID that maps sandbox pod
annotations to spiffe://openshell.local/openshell/sandbox/<sandbox-id>.
OpenShell mounts the SPIFFE CSI Workload API socket at
/spiffe-workload-api/spire-agent.sock into sandbox pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
ServiceAccount bootstrap and gateway-minted sandbox JWT path.
Cluster Lifecycle (suspend/resume)
Stop the cluster without losing state (faster than delete/recreate):
mise run helm:k3s:stop
mise run helm:k3s:start
Check cluster status:
mise run helm:k3s:status
Helm Chart Checks
Run the chart lint task before changing Helm templates, values overlays, or Skaffold inputs:
mise run helm:lint
If Helm reports missing chart dependencies, remove the specific stale subchart
archive or directory named by the error from deploy/helm/openshell/charts/,
then rerun the lint task.
For example, when lint reports chart metadata is missing these dependencies: postgresql, remove stale PostgreSQL chart artifacts:
rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint
The charts/ directory is ignored and regenerated by helm dependency build
for dependencies still declared in Chart.yaml.
Key Files
| Path | Purpose |
|---|---|
deploy/helm/openshell/skaffold.yaml | Skaffold config — images, Helm releases, values overlays |
deploy/helm/openshell/values.yaml | Default Helm values |
deploy/helm/openshell/ci/values-skaffold.yaml | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
deploy/helm/openshell/ci/values-cert-manager.yaml | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
deploy/helm/openshell/ci/values-gateway.yaml | Envoy Gateway GRPCRoute + Gateway overlay |
deploy/helm/openshell/ci/values-high-availability.yaml | HA test overlay (replicaCount: 2 with external PostgreSQL Secret) |
deploy/helm/openshell/ci/values-keycloak.yaml | Keycloak OIDC overlay |
deploy/helm/openshell/ci/values-sidecar.yaml | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
deploy/helm/openshell/ci/values-spire.yaml | SPIFFE/SPIRE provider token grant overlay |
deploy/helm/openshell/ci/values-spire-stack.yaml | SPIRE hardened chart values for local dev |
deploy/helm/openshell/ci/values-tls-disabled.yaml | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
deploy/kube/manifests/envoy-gateway-openshell.yaml | GatewayClass for Envoy Gateway (mise run helm:gateway:apply) |
tasks/scripts/helm-k3s-local.sh | k3d cluster create/delete/start/stop/status |
tasks/scripts/keycloak-k8s-setup.sh | Keycloak deploy + realm import |