helm-dev-environment

Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).

Prerequisites

• Docker Desktop (macOS) or Docker Engine (Linux) running
• mise install completed (provides k3d, kubectl, skaffold, helm)

Startup

1. Create the cluster

mise run helm:k3s:create

Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file. Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox releases) and preloads the default community sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time.

Multi-worktree support: the cluster name is derived from the last component of the current git branch (e.g. branch kube-support/local-dev/tmutch → cluster openshell-dev-tmutch). Each worktree therefore gets its own isolated cluster and its own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

Override with env vars before running helm:k3s:create:

• HELM_K3S_LB_HOST_PORT (default: 8080)
• HELM_K3S_PRELOAD_SANDBOX_IMAGE (default: ghcr.io/nvidia/openshell-community/sandboxes/base:latest; set to an empty value to skip)

2. Deploy OpenShell

Iterative dev (rebuilds on file changes, recommended during active development):

mise run helm:skaffold:dev

One-shot deploy (build once and leave running):

mise run helm:skaffold:run

Supervisor sidecar topology (build once and leave running):

mise run helm:skaffold:run:sidecar

Supervisor sidecar topology with TLS/mTLS enabled (build once and leave running):

mise run helm:skaffold:run:sidecar-mtls

Both commands build the gateway and supervisor images and deploy the OpenShell Helm chart. The sidecar profile renders an openshell-network-init init container for nftables setup and an openshell-supervisor-network runtime sidecar for proxying. Binary-aware policy mode runs that sidecar as UID 0 with SYS_PTRACE and DAC_READ_SEARCH; relaxed mode can run it as the configured proxy UID, which must be at least 1000 and distinct from the workload UID. The sidecar-mTLS profile reuses ci/values-sidecar.yaml and restores server.disableTls=false inline for Skaffold. The pkiInitJob hook (a pre-install Job that runs openshell-gateway generate-certs) generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or kubectl port-forward.

HA test deploy (two gateway replicas + external PostgreSQL Secret): uncomment #- ci/values-high-availability.yaml in deploy/helm/openshell/skaffold.yaml, create the Secret named openshell-ha-pg with a uri key, then run mise run helm:skaffold:run or mise run helm:skaffold:dev.

TLS behaviour

ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run plaintext by default. To test sidecar topology with TLS enabled, use mise run helm:skaffold:run:sidecar-mtls.

Connecting via port-forward

Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so the port-forward uses local port 8090 to avoid a collision:

KUBECONFIG=kubeconfig kubectl port-forward -n openshell svc/openshell 8090:8080

Plaintext (default Skaffold deploy):

openshell sandbox list --gateway-endpoint http://localhost:8090

With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:

mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key

The server cert SANs include localhost and 127.0.0.1, so hostname verification passes over a port-forward without any extra flags:

openshell sandbox list --gateway-endpoint https://localhost:8090

Teardown

Remove the Helm releases (keep cluster)

mise run helm:skaffold:delete

For a sidecar-profile deployment:

mise run helm:skaffold:delete:sidecar

Delete the cluster entirely

mise run helm:k3s:delete

This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.

Optional Add-ons

Each add-on requires uncommenting the corresponding valuesFiles entry in deploy/helm/openshell/skaffold.yaml before running helm:skaffold:dev or helm:skaffold:run.

Envoy Gateway (Gateway API / GRPCRoute)

Envoy Gateway is already installed by Skaffold (the envoy-gateway Helm release in skaffold.yaml). To activate routing:

1. Uncomment #- values-gateway.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run
3. Apply the GatewayClass: mise run helm:gateway:apply
4. Access: http://127.0.0.1:8080

values-gateway.yaml creates a Gateway (listener on port 80, class eg) and a GRPCRoute in the openshell namespace. Envoy Gateway provisions a LoadBalancer service for the proxy; klipper-lb binds it to hostPort 80, reachable via the 8080:80 load balancer port mapping.

Keycloak OIDC

One-time setup — only needed once per cluster lifetime:

mise run keycloak:k8s:setup

This deploys Keycloak (quay.io/keycloak/keycloak:24.0) into the keycloak namespace, imports the openshell realm from scripts/keycloak-realm.json, and prints a port-forward command for acquiring tokens from the CLI.

Then activate OIDC in the OpenShell Helm chart:

1. Uncomment #- ci/values-keycloak.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run

To remove Keycloak:

mise run keycloak:k8s:teardown

SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate SPIFFE JWT-SVIDs for dynamic provider token grants:

1. Uncomment the spire-crds and spire releases in deploy/helm/openshell/skaffold.yaml
2. Uncomment #- ci/values-spire.yaml in the OpenShell release values files
3. Redeploy: mise run helm:skaffold:run

ci/values-spire-stack.yaml configures the local SPIRE trust domain as openshell.local and adds a ClusterSPIFFEID that maps sandbox pod annotations to spiffe://openshell.local/openshell/sandbox/<sandbox-id>. OpenShell mounts the SPIFFE CSI Workload API socket at /spiffe-workload-api/spire-agent.sock into sandbox pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes ServiceAccount bootstrap and gateway-minted sandbox JWT path.

Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):

mise run helm:k3s:stop
mise run helm:k3s:start

Check cluster status:

mise run helm:k3s:status

Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or Skaffold inputs:

mise run helm:lint

If Helm reports missing chart dependencies, remove the specific stale subchart archive or directory named by the error from deploy/helm/openshell/charts/, then rerun the lint task.

For example, when lint reports chart metadata is missing these dependencies: postgresql, remove stale PostgreSQL chart artifacts:

rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint

The charts/ directory is ignored and regenerated by helm dependency build for dependencies still declared in Chart.yaml.

Key Files