k8s-launch-kit-discover

l8k: Cluster Discovery

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

This workflow uses the default host target; --target host is equivalent. The CLI snapshots discovery arguments and runs the concrete Host discovery operation through the target registry; command syntax and artifacts are unchanged.

Discover cluster hardware and produce a cluster-config.yaml describing NICs, GPUs, rails, and node groups. Use docs/user/profiles.md#check-prerequisites-and-site-inputs for the site-owned decisions and docs/user/discovery.md#review-saved-inventory for the post-discovery review before generation.

Three things to know about the saved file:

• Fresh discovery persists a resolved profile. Without --user-config, discovery fills profile fields from hardware/defaults and applies explicit CLI overrides last. With --user-config, it replaces only clusterConfig, preserves every other section, and then applies explicit CLI overrides.
• North-south-only groups are dropped. A node group whose NICs are all north-south (BlueField DPUs / OOB NICs) produces no manifests and is omitted from cluster-config.yaml (it would otherwise consume an NV-IPAM subnet slice and shift the per-rail pools). North-south NICs alongside east-west NICs in a mixed group are still listed.
• Documentation comments are preserved. The field-reference comments from the source config (l8k-config.yaml or --user-config) are carried over into the saved cluster-config.yaml, with a Generated by l8k discover banner at the top.

l8k discover is self-contained. It does NOT require a pre-installed Network Operator. On every run it bootstraps the NIC Configuration Daemon (and the 5 nic-configuration-operator CRDs, only if missing) into a private namespace nvidia-k8s-launch-kit, reads NicDevice CRs published by the daemon, then tears the namespace down on exit (unless --keep-namespace is set). The daemon image is pulled from <networkOperator.repository>/ nic-configuration-operator-daemon:<networkOperator.componentVersion> (both fields come from l8k-config.yaml).

Usage (from AI agent)

l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json >discovery.json 2>discovery.log &&
  jq . discovery.json

Check the l8k exit status before parsing; retain discovery.log on failure.

Usage (human-interactive)

l8k discover --save-cluster-config <OUTPUT> [--kubeconfig <PATH>]

Flags

Examples

# Basic discovery
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Using $KUBECONFIG env var (no --kubeconfig needed)
l8k discover --save-cluster-config ./cluster-config.yaml

# Keep the bootstrap namespace for debugging
l8k discover \
  --kubeconfig ~/.kube/config \
  --keep-namespace \
  --save-cluster-config ./cluster-config.yaml

# Refresh clusterConfig in an existing config
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Persist explicit profile overrides
l8k discover \
  --kubeconfig ~/.kube/config \
  --fabric infiniband --deployment-type rdma_shared \
  --multirail=false \
  --save-cluster-config ./cluster-config.yaml

# Agent mode (JSON output)
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json >discovery.json 2>discovery.log

Output Format

The generated cluster-config.yaml contains the final profile plus a clusterConfig[] hardware-group array:

profile:
  fabric: ethernet
  deployment: sriov
  multirail: true
clusterConfig:
  - identifier: "dgx-b200-h100-nvl"
    machineType: DGX-B200
    gpuType: NVIDIA-H100-NVL
    netplanManaged: true
    capabilities:
      nodes:
        sriov: true
        rdma: true
    pfs:
      - deviceID: "101e"
        networkInterface: "eth0"
        rail: 0
    workerNodes: [node-01, node-02]
    nodeSelector:
      nvidia.kubernetes-launch-kit.machine: "dgx-b200-h100-nvl"
    thirdPartyRDMAModules: [nv_peer_mem]

Discovery patches every node in the group with two labels:

• nvidia.kubernetes-launch-kit.machine: <identifier> — per-source-group identity, identical to the persisted group identifier and used as its nodeSelector.
• nvidia.kubernetes-launch-kit.gpu: <gpuType> — used as the merged-group nodeSelector when l8k generate auto-merges source groups sharing a GPU type.

Group identifiers and machine-label values are lowercase and omit complete NVIDIA segments. The same pass maps ThinkSystem to ts and PowerEdge to pe. They are bounded to 30 bytes; long values retain balanced machine/GPU prefixes plus a 6-character deterministic hash, and unused prefix space moves to the longer component. GPU-only labels retain their discovered case and vendor segment, use Kubernetes' 63-byte limit, and keep the existing 8-character shortening hash. When machineType or gpuType couldn't be resolved (GPU operator labels absent and hardware probe failed), a fallback group-N identifier is used and the machine label is not written; the GPU label is still written when gpuType alone is resolved.

Prerequisites

• Node Feature Discovery (NFD) is not required. Before bootstrapping the daemon, discovery lists Nodes and renders node-name affinity for nodes that are Ready=True and not spec.unschedulable. The DaemonSet still has no NFD nodeSelector and tolerates taints, so eligible control-plane/tainted nodes are included. NIC-bearing nodes are detected by a sysfs probe for PCI vendor 0x15b3 rather than the NFD feature.node.kubernetes.io/pci-15b3.present label. BlueFields for which mlxprivhost reports level: restricted are excluded because the daemon will not publish NicDevice resources for them. Mixed nodes remain eligible when they have at least one non-restricted NVIDIA NIC.
• Image-pull access from every worker node to <networkOperator.repository>/nic-configuration-operator-daemon:<networkOperator.componentVersion>. Use --image-pull-secrets <name> (or networkOperator.imagePullSecrets in the config file) for private registries.
• Network Operator does NOT need to be pre-installed.

Fresh discovery writes validation.gpuDirect.enabled. It is true only when every worker in every discovered group can satisfy its render bucket's topology-derived validation.gpuDirect.gpuResourceType request; otherwise it is false. With --user-config, the supplied validation section is preserved. Do not add GPU counts or resource maps under clusterConfig.

Tips

• Fresh profile precedence is hardware/default < explicit CLI. On a rerun with --user-config, no profile fields are recalculated; explicit CLI flags still override their corresponding values.

• A fresh discovery ignores the example profile in the installed reference config. Supplying that file explicitly through --user-config makes its profile values user input and therefore preserves them.

• If fabric probes are mixed or unconfirmed, discovery still writes the file with an empty profile.fabric; set --fabric on discovery or generation.

• clusterConfig[].netplanManaged is true when any worker has an NVIDIA PF selected by a host netplan match.macaddress plus set-name rule. Discovery checks every worker and uses PF MACs only transiently; they are not saved in the shared group config. If generation would emit a NicInterfaceNameTemplate for an affected group, clean up the host's set-name stanzas and re-run discovery before retrying generation; do not clear the field manually.

• The bootstrap namespace is always nvidia-k8s-launch-kit — not configurable. The daemon's SA / ClusterRole / ClusterRoleBinding are renamed to k8s-launch-kit-nic-config-daemon so they don't collide with the cluster-scoped names a coexisting Network Operator install would create.

• CRDs (nicdevices, nicconfigurationtemplates, etc.) are applied only when missing. If they already exist (because Network Operator or a prior l8k discover run created them), they're left alone — discovery never overwrites a different version.

• After discovery finishes the namespace is torn down (cascade-delete handles SA / RoleBinding / DaemonSet / pods / NicDevice CRs); the CRDs intentionally persist so any external NicDevice consumers survive. Cluster-scoped ClusterRole / ClusterRoleBinding are deleted explicitly.

• Pass --keep-namespace to leave the namespace in place — useful when debugging daemon pod start-up failures (kubectl describe pod -n nvidia-k8s-launch-kit).

• Before bootstrapping, discovery pre-cleans the nvidia-k8s-launch-kit namespace (deletes any leftover DaemonSet/pods/NicDevice CRs from a crashed prior run and waits up to 2 min for it to clear) so a fresh daemon is never layered on stale state.

• Discovery waits up to 5 minutes for daemon pods to be Ready, but tolerates stuck pods: if some pods are wedged (e.g. ImagePullBackOff/CrashLoopBackOff on an unrelated node) it proceeds with the Ready ones rather than blocking the whole window, and only aborts if no pod ever becomes Ready. Common causes of a total failure: image tag missing in your registry, pull-secret missing.

• If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found", the daemon ran but no node's sysfs exposed a 0x15b3 device (no Mellanox/NVIDIA NICs, or /sys not mounted/readable in the pod).

• If discovery reports "no discoverable NVIDIA NICs were found", every detected BlueField is in zero-trust (restricted) mode. Those devices cannot be configured from the host and intentionally do not publish NicDevice resources.

• After determining each group's (machineType, gpuType), discovery looks up a topology preset under presets/ using exact-match lookup on that pair. A matching preset overrides heuristic-derived topology fields (traffic class, rail, NUMA, GPU affinity). There is no any-GPU fallback — a preset with empty gpuType: is rejected at load time. If no preset matches, discovery proceeds with heuristic classification.

• If you already know the SKU and want to skip cluster discovery entirely, use l8k generate --for <preset> (see k8s-launch-kit-generate).

Rail collapsing (one rail per NIC)

By default (--collapse-nic-rails, default true) discovery advertises one rail per NIC, not one per PF. A single NIC can expose several east-west PFs that are planes of one physical port (Spectrum-X multi-plane ConnectX-8/9). For those, only the master PF (lowest PCI function) is written to cluster-config.yaml, so the rail count reflects physical NICs — e.g. an 8-PF ConnectX-8 node yields 4 rails, not 8. The per-PF planes are reconstructed at generate time from the multiplane mode + numberOfPlanes.

The exception is a genuinely dual-port NIC: when the NIC's VPD model name (read from NicDevice.Status.modelName, surfaced as the model field on each PF) contains a port-count keyword like 2-port/Dual-port, each port is kept as its own rail. Single-port (1P) and multi-plane models (no port-count keyword) collapse. An empty/unreadable model collapses by default.

Pass --collapse-nic-rails=false to restore the legacy one-rail-per-PF behaviour (handy on dev setups). Note: a collapsed group no longer has the same PF count as a full-PF topology preset, so such presets won't apply by exact match and the live (collapsed) classification is kept (a preset-deviation warning is emitted).

See Also

• k8s-launch-kit-shared — Global flags and output modes
• k8s-launch-kit-config — Understand the config file discovery produces
• k8s-launch-kit-generate — Generate manifests from cluster config

OpenShift

On OpenShift, pass --flavor ocp. Use explicit clusterConfig[].workerNodes in --user-config to bound discovery; the temporary discovery ServiceAccount gets privileged SCC use in its own namespace. Follow the complete prerequisite and review sequence in docs/user/openshift.md.

RA2.4 integration

RA2.3 is limited to Network Operator 26.7; RA2.4 defaults to 26.10 and accepts newer catalogued releases. RA2.4 needs a full doSPCX ConfigMap, preserving binaryData and annotations and rendering in the resolved operator namespace. 26.10 beta.2 lacks the packaged NCO platformType CRD: verify a compatible NCO implementation and matching CRDs before deployment. Deploy checks the served NCT schema after policy bootstrap and blocks additional resource apply when platformType support is absent or cannot be inspected. External Helm, overwrite, and dry-run do not bypass this check; dry-run needs APIs installed already.

Derived clusterConfig[].spectrumX.platformType uses the longest case-insensitive substring from the internal doSPCX platform list in gpuType. No preset or mapping setting is needed. Discovery saves empty values and warns; generation rejects unresolved selected RA2.4 groups. RTX/CX8 defaults to none/one plane. Mapping does not itself qualify hardware. Optional spectrumX.ovsConfig is a string map; clusterConfig[].spectrumX.swPlaneByRail assigns existing HWPLB rails, defaults to zero, and must agree across merged groups. Refresh preserves explicit plane assignments only for unchanged source/rail hardware and verified unchanged worker membership (ordering does not matter). Changed or unknown membership drops prior assignments with a review/reapply warning. Every selected RA2.4 source needs complete dense rail IDs, uniform NICs per rail, and a divisible planes/NIC ratio; merged sources must have the same effective layout. Merged GPU selectors must not include excluded same-GPU sources in other buckets; select one source in that case. One physical NIC cannot span rails (the current ThinkSystem-SR650-V4-RTX-PRO-6000 preset has this unsupported layout). Multiple hardware partitions inside one logical rail are outside this layout. Feature gates are not added.