k8s-launch-kit-config

l8k: Configuration Files

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

Understand, create, or edit l8k configuration files.

File Types

Usage

# Use a config file for generation
l8k generate --user-config my-config.yaml \
  --save-deployment-files ./output

# Combine user config with live discovery (hardware refreshed, settings kept)
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./updated-config.yaml

Profile Resolution and Write-Back

Discovery and file-backed generation resolve and persist settings with this precedence:

1. Hardware and built-in defaults fill missing fields.
2. Values already present in --user-config are preserved.
3. Explicit CLI flags override both.

Discovery writes to its selected output YAML; generation rewrites its source config in place. On later runs, only missing fields are recalculated. multirail: false is an explicit value, not a missing field, so it remains false across rewrites.

Config Sections Quick Reference

Each clusterConfig[] entry has these key fields:

• identifier — group name (used for NicNodePolicy naming). For groups with both machineType and gpuType resolved, this is the sanitised machine label (<machineType>-<gpuType>); otherwise a fallback group-N.
• machineType — server model (e.g. PowerEdge-XE9680); populated from nvidia.com/gpu.machine label or DMI fallback.
• gpuType — GPU SKU (e.g. NVIDIA-H200); populated from nvidia.com/gpu.product label or nvidia-smi fallback. Note: this field used to be called productType — the rename happened to disambiguate it from the server model. Old productType: keys in hand-authored configs must be renamed to gpuType:.
• capabilities.nodes.{sriov,rdma,ib} — what the underlying hardware supports.
• pfs[] — physical function list with PCI address, device ID, RDMA device, network interface, traffic class, rail, NUMA, GPU affinity, and model (the VPD model/description string read from NicDevice.Status.modelName).
  • One rail per NIC (default). Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose model is genuinely dual-port (2-port/Dual-port) keeps a rail per port. Run l8k discover --collapse-nic-rails=false to emit one rail per PF (legacy/dev behaviour).
• nodeSelector — Kubernetes node selector for this group. Source groups key on the machine label written by l8k discover: nvidia.kubernetes-launch-kit.machine: <machineType>-<gpuType>. Auto-merged groups (different machineTypes sharing a GPU type) key on nvidia.kubernetes-launch-kit.gpu: <gpuType> instead — discovery writes both labels onto every node, so the merged selector binds correctly across source machineTypes.
• workerNodes — explicit hostnames (populated by discovery).

For the full field-by-field reference with types, defaults, and descriptions, read references/config-reference.md.

Topology Presets

The presets/ directory contains pre-recorded topologies for known hardware combinations. A preset is a topology.yaml file with the following shape:

machineType: PowerEdge-XE9680   # required
gpuType: NVIDIA-H200            # required — both keys are matched as a pair
nicModel: BlueField-3 SuperNIC (ConnectX-7)
gpuInterconnect: NV18
numaNodes: 2

# Required if the preset will be used with `l8k generate --for`.
# Discovery-time overlay does not need this block.
capabilities:
  nodes:
    sriov: true
    rdma: true
    ib: false

pfs:
  - deviceID: a2dc
    pciAddress: 0000:1a:00.0
    traffic: east-west
    rail: 0
    numaNode: 0
    connectedGPU: GPU0
    gpuProximity: PIX

Lookup is exact-match on (machineType, gpuType). No any-GPU fallback — a preset that doesn't declare gpuType: is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like PowerEdge-XE9680-H200 / PowerEdge-XE9680-B200. The directory name is shown by l8k preset list and is what l8k generate --for <name> accepts.

Validation deviations. When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is NOT applied — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under clusterConfig[*].presetDeviation and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid only on an exact match (zero deviations); presetApplied: true appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application.

Common Edits

# Change VF count per PF
sriov:
  numVfs: 16

# Change MTU
sriov:
  mtu: 9000

# Set DOCA driver version
docaDriver:
  version: "doca3.2.0-25.10-1.2.8.0-2"

# Allow four simultaneous maintenance operations. Network Operator 26.1+
# uses the global Maintenance Operator limits; older releases use the legacy
# SR-IOV/OFED limits where applicable.
maintenance:
  maxParallelOperations: 4
  maxUnavailable: 4
  maxNodeMaintenanceTimeSeconds: 3600
  maxParallelUpgrades: 4

# Override the helm chart repository URL (rarely needed — the embedded
# release catalog supplies the right URL for each MAJOR.MINOR release).
# Useful only for mirrors or private chart hosts.
networkOperator:
  helmRepoURL: "https://my-mirror.example.com/charts"

# Configure NV-IPAM subnets manually
nvIpam:
  subnets:
    - name: "rail-0-subnet"
      cidr: "10.10.0.0/16"
      gateway: "10.10.0.1"

# Namespaces for the secondary-network CRs + example test DaemonSets.
# One independent copy is rendered per namespace (shared resources like
# IPPools and NodePolicies are NOT duplicated). Defaults to ["default"].
networkNamespaces: ["my-namespace"]

Tips

• Start by running discovery (l8k discover) to generate a baseline, then edit it.
• A discovered config can be passed directly to l8k generate without repeating profile flags; use flags only for overrides.
• nvIpam subnets are auto-generated if not specified — one per rail using non-routable ranges.
• docaDriver.unloadThirdPartyRDMAModules: true auto-populates UNLOAD_THIRD_PARTY_RDMA_MODULES from discovered OFED-dependent modules.
• For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
• Updating an existing release to the generated requestor-mode Helm values requires --overwrite-existing.

See Also

• k8s-launch-kit-shared — Global flags
• k8s-launch-kit-discover — Generate a config from live cluster
• references/config-reference.md — Complete annotated YAML, including maintenance value restrictions