aicr-analyzing-snapshots

Analyzing AICR Snapshots

Systematic analysis of AICR snapshot YAML files to extract cluster identity, provider characteristics, GPU topology, node health, software stack, and operational signals. Produces a structured Markdown report.

When to Use

• User provides a snapshot YAML file for review
• User asks about cluster characteristics or provider differentiation
• User wants to compare snapshots or extract specific insights
• User asks to generate a cluster assessment report

Analysis Procedure

Snapshot files are large (50K-80K+ tokens). Never read the whole file. Use mcp__plugin_context-mode_context-mode__execute_file with Python/YAML parsing to extract sections, or use targeted Read with offset/limit on specific line ranges found via Grep.

Step 1: Extract Metadata and Structure

import yaml
data = yaml.safe_load(FILE_CONTENT)
meta = data.get('metadata', {})
measurements = data.get('measurements', [])
print("=== METADATA ===")
for k, v in meta.items():
    print(f"  {k}: {v}")
print("\n=== MEASUREMENTS ===")
for m in measurements:
    subtypes = [s.get('subtype', s.get('name', '?')) for s in m.get('subtypes', [])]
    print(f"  {m['type']}: {subtypes}")

Step 2: Extract K8s Server and Node Info

Key fields for provider identification:

Field Path	What It Reveals
K8s.server.version	K8s version + vendor suffix (-eks-, -gke, -aks, +lke)
K8s.node.provider	Mapped provider: eks, gke, aks, oke, lke, metal3, kind
K8s.node.provider-id	Raw provider URI (aws://, gce://, azure://, oci://, linode://, metal3://)
K8s.node.kernel-version	Kernel + arch indicator (e.g., -64k = ARM 64K pages)
K8s.node.container-runtime-*	Runtime name and version
K8s.node.kubelet-version	Kubelet version
K8s.node.os-image	OS description string

Provider detection logic:

provider-id prefix	Service	Notes
aws://	eks	Amazon EKS
gce://	gke	Google GKE
azure://	aks	Azure AKS
oci://	oke	Oracle OKE
linode://	lke	Akamai Cloud / Linode LKE
metal3://	bare-metal	Metal3/Ironic, self-managed
kind://	kind	Local dev cluster
(none/other)	any	Self-managed, check version string

If provider-id is absent, check K8s.server.version for vendor substrings.

Step 3: Extract GPU Info

Key fields from GPU.smi:

Field	Example	Significance
gpu.model	NVIDIA GB300	Maps to accelerator criteria
gpu.product-architecture	Blackwell	GPU generation
gpu-count	4	GPUs per node
driver	580.126.16	NVIDIA driver version
cuda-version	13.0	CUDA toolkit version
gpu.addressing-mode	ATS	ATS = unified CPU-GPU memory (Grace)
gpu.persistence-mode	Disabled/Enabled	Should be Enabled for production
gpu.vbios-version	97.10.4A.00.1A	Firmware version
gpu.gsp-firmware-version	580.126.16	GSP firmware

Accelerator mapping (checked in order, case-insensitive):

gpu.model contains	Accelerator
gb200	gb200 (check before b200)
gb300	gb200 class (Blackwell NVL family)
b200	b200
h100	h100
gh200	unresolved — Grace Hopper Superchip, not the discrete H200 GPU (check before h200)
h200	h200 (discrete H200 GPU)
a100	a100
l40s	l40s
l40	l40
rtx pro 6000	rtx-pro-6000

Step 4: Extract OS Info

From OS.release: ID, VERSION_ID, PRETTY_NAME

From OS.grub: Boot parameters (check for iommu, console, init_on_free)

From OS.kmod: Loaded kernel modules (look for nvidia*, nv_peer_mem, gdrdrv, ib_*, mlx5_* for RDMA/InfiniBand)

From OS.sysctl (key tuning parameters):

Sysctl	Good Value for GPU	Why
vm.swappiness	<= 10	Minimize swapping for GPU workloads
vm.overcommit_memory	1	Allow overcommit for training
vm.nr_hugepages	> 0 (ideal)	Large page performance
fs.file-max	High (9223372036854775807)	Sufficient file descriptors
kernel.threads-max	> 1M	Sufficient threads
vm.min_free_kbytes	> 1M	Memory reserve

Step 5: Extract Node Topology

From NodeTopology.summary: node-count, taint-count, label-count

From NodeTopology.taint and NodeTopology.label, read the items list — one entry per distinct reading, sorted by key/value (taints: key/effect/value):

Item Field	What It Holds
context.key	Taint or label key, verbatim
context.value	Taint or label value (may be empty)
context.effect	Taints only: NoSchedule, PreferNoSchedule, NoExecute
data.node-count	True node total, including nodes dropped by truncation
data.node-list	Comma-separated node names (one of node-list / node-list-ref)
data.node-list-ref	Key into the subtype's data map whose entry holds the names (one of node-list / node-list-ref)
data.truncated	true when the node list is capped and ends with (+N more)

Current snapshots also carry the older data map on both subtypes; items is authoritative. Take counts from data.node-count rather than splitting node-list, and read data.truncated rather than probing for a (+N more) suffix. Use topology.LabelReadings / TaintReadings to resolve items into hydrated readings — they expand node-list-ref automatically, so callers do not need to implement the reference logic themselves.

Older snapshots (no items): fall back to the folded data map — effect|value|node1,node2,... for taints, value|node1,node2,... for labels. That encoding is lossy, so qualify anything derived from it:

• A map key is ambiguous: when a key carries more than one value the value is folded into the key as <key>.<value>, indistinguishable from a label literally named that, and one of the colliding readings is dropped. Report such a key verbatim instead of asserting a key/value split.
• A taint key disambiguated the same way ends in .<effect> and its value has only two fields (value|nodes); two taints sharing key and effect collapse into one entry.
• summary.taint-count / label-count count map entries there, so they under-report wherever a collapse occurred, and node counts reflect only what survived truncation.

High-value labels to extract (skip feature.node.kubernetes.io/cpu-cpuid.*):

Label Prefix	What It Reveals
kubernetes.io/arch.*	CPU architecture (amd64 vs arm64 = heterogeneous)
nvidia.com/gpu.*	GPU product, family, memory, compute, count, MIG state
nvidia.com/cuda.*	CUDA driver/runtime versions
nvidia.com/mig.*	MIG capable/config/strategy
nvidia.com/gpu.clique.*	NVLink GPU cliques (multi-node NVLink domains)
resource.nvidia.com/computeDomain	Unified compute domain
network.topology.nvidia.com/accelerator.*	NVLink fabric blocks
node-type.*	Hardware type (gb300, standard)
node-pool.*	Pool assignment (gpu-pool, cpu-pool)
node.dgxc.nvidia.com/*	DGX Cloud node classification
k8saas.nvidia.com/*	K8SaaS management (NVSentinel cordon/uncordon)
dgxc.nvidia.com/nvsentinel-state	Health state (remediation-failed, healthy)
nvsentinel.dgxc.nvidia.com/*	NVSentinel component versions, driver state
network.nvidia.com/operator.*	Network operator MOFED/NIC config state
metal3.io/uuid.*	Metal3 bare-metal node UUIDs
workload.*	Workload type (gpu, general)
feature.node.kubernetes.io/rdma.*	RDMA available/capable
feature.node.kubernetes.io/network-sriov.*	SR-IOV capability
feature.node.kubernetes.io/pci-15b3.*	Mellanox ConnectX presence
feature.node.kubernetes.io/pci-10de.*	NVIDIA GPU PCI presence
nvidia.com/dra-kubelet-plugin	DRA (Dynamic Resource Allocation)

Step 6: Extract K8s Images and Policies

From K8s.image: All deployed container images and versions.

From K8s.policy: Flattened GPU Operator ClusterPolicy spec (dot-notation).

Key policy fields:

Policy Field	What to Check
driver.enabled	GPU driver managed by operator
driver.version	Driver version in policy
driver.rdma.enabled	RDMA support
toolkit.enabled	Container toolkit
devicePlugin.enabled	Device plugin active
dcgm.enabled / dcgmExporter.enabled	GPU monitoring
migManager.enabled	MIG management
ccManager.enabled / ccManager.defaultMode	Confidential Computing
sandboxWorkloads.enabled	Sandbox/KubeVirt workloads
psa.enabled	Pod Security Admission
vfioManager.enabled	VFIO passthrough

Step 7: Extract Slinky and MariaDB Conflict Signals

From K8s.slinky-slurm, report:

• collection-state: absent, detected, unsupported-multicluster, or unknown
• Controller count and projected NodeSet/LoginSet/RestApi/Accounting counts
• Item identities and Controller associations; include only the allowlisted item data already present in the snapshot

detected means a Controller declaration exists, not that Slurm or its operator is healthy. Child items and counts are emitted only after all required APIs and references are collected conclusively; their absence is otherwise not confirmed absence. Never infer platform: slurm from this subtype.

From K8s.mariadb-operator, report collection-state as official MariaDB-operator API conflict evidence:

• absent: official API group conclusively absent
• api-detected: official API footprint present without observed MariaDB CRs
• crs-detected: one or more official MariaDB CRs observed
• unknown: discovery or List was inconclusive

These states do not prove database availability, operator health, or the existence of an external database such as RDS. Never infer accounting.databaseSource.

Step 8: Check SystemD Services

From SystemD.containerd.service, SystemD.kubelet.service, SystemD.docker.service:

Field	What to Check
ActiveState	Should be active
SubState	Should be running
LimitNOFILE	File descriptor limits
LimitMEMLOCK	Memory lock limits (important for RDMA)
KillMode	process for containerd (graceful)
Delegate	true for containerd (cgroup delegation)
CPUAccounting	Resource accounting

Report Template

Structure the output as:

# Snapshot Analysis: {name}
> Source: {file} | Captured: {timestamp} | AICR: {version}

## Cluster Identity
Table: source-node, provider, K8s version, node count, GPU model, total GPUs

## Provider-Differentiating Insights
### 1. Provider Type (cloud vs bare-metal, managed vs self-managed)
### 2. CPU Architecture (homogeneous vs heterogeneous, ARM vs x86)
### 3. GPU Hardware (model, architecture, memory, driver, CUDA, MIG, persistence)
### 4. Network Topology (NVLink blocks, cliques, compute domains, RDMA, SR-IOV)
### 5. Management Layer (K8SaaS, NVSentinel health, cordon state)
### 6. Job Scheduling (Slurm/Slinky presence, HPC vs cloud-native)
### 7. Networking Stack (CNI, RDMA, SR-IOV, DOCA/MOFED)
### 8. Security (Confidential Computing, PSA, DRA)
### 9. Operational Signals (sysctl tuning, hugepages, persistence mode)

## Software Stack
### Key Container Images (table)
### OS and Kernel (table)

## Node Inventory
List nodes by rack/block/pool

## Operational Flags
Anything unusual: GPU health issues, disabled persistence mode,
missing hugepages, NVSentinel remediation failures, etc.

What Makes Each Provider Unique

Cloud Providers (EKS, GKE, AKS, OKE)

• Provider-id with cloud prefix
• Cloud-specific K8s version suffixes
• Managed node groups / auto-scaling
• No bare-metal labels (metal3.io)
• Typically x86_64 homogeneous
• No NVLink fabric topology labels
• No Slurm/Slinky stack

Bare-Metal / DGX Cloud (Metal3, K8SaaS)

• metal3:// provider-id with per-node UUIDs
• k8saas.nvidia.com/* management labels
• NVSentinel health monitoring (cordon/uncordon lifecycle)
• NVLink accelerator blocks and GPU cliques
• Compute domains spanning racks
• ARM64 Grace CPUs (heterogeneous with x86 head node)
• Slurm/Slinky HPC scheduling
• RDMA + SR-IOV networking with DOCA drivers
• ATS GPU addressing mode (unified memory)
• Liquid-cooled chassis machine types (LCC in machine name)

Self-Managed / Kind

• Missing or generic provider-id
• No cloud or bare-metal management labels
• Simpler topology (single node or small cluster)
• Standard x86_64

AICR Criteria Mapping

After analysis, map the snapshot to AICR recipe criteria:

aicr recipe \
  --service {detected_service} \
  --accelerator {detected_accelerator} \
  --os {detected_os} \
  --intent {training|inference} \
  --snapshot {snapshot_file}

Criteria	Extracted From	Valid Values
service	K8s.node.provider / K8s.server.version	eks, gke, aks, oke, kind, lke
accelerator	GPU.smi.gpu.model	h100, h200, gb200, b200, a100, l40s, l40, rtx-pro-6000
os	OS.release.ID	ubuntu, rhel, cos, amazonlinux, talos, ol
intent	User-specified	training, inference
platform	User-specified	dynamo, kubeflow, nim, runai, slurm