backdoor-deployment

Backdoor Deployment Automation

Validates a container image change by deploying the current production image, collecting baseline data, then deploying the test image (from a CI build) and comparing data volume and resource consumption. No regressions = pass.

Required Inputs

Only these two come from the user. Everything else is derived — do not ask for it.

Derived Values

Resolve all of these automatically.

The chart carries no cluster coordinates. charts/azuremonitor-containerinsights is the same chart CI/CD deploys, and its values-template.yaml ships placeholders (<your_cluster_id>, <your_workspace_id>, …) rather than real values, so nothing can be parsed out of it. The cluster resource ID comes from the user and the workspace is resolved from the DCR; both are passed as --set overrides.

Build Pipeline

General Rules

• Save the output of each step to BackdoorDeploymentOutput.md in the repo root. Always append new results at the end. Beautify for readability. Don't clear until explicitly asked.
• If asked "what's the next step", read BackdoorDeploymentOutput.md and suggest the next step.
• Before executing any step, verify previous step data exists in BackdoorDeploymentOutput.md. If missing, confirm with the user before proceeding.
• If the build must be retriggered, keep the existing production baseline data — do not re-deploy the production image or re-collect baseline data.
• After the workflow completes, redeploy the production image so the cluster is left on prod, and re-enable the managed addon if you disabled it. The generated Chart.yaml / values.yaml are gitignored and may be left in place or deleted — either way the tree stays clean.

Procedures

Resolve the workspace

Never ask the user for the workspace. Derive it from the cluster's ContainerInsightsExtension DCR association, which is the workspace the cluster actually ships data to:

AKS_RESOURCE_ID="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/<name>"

DCR_ID=$(az monitor data-collection rule association list --resource "$AKS_RESOURCE_ID" \
  --query "[?name=='ContainerInsightsExtension'].dataCollectionRuleId | [0]" -o tsv)

WS_RESOURCE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \
  --query "destinations.logAnalytics[0].workspaceResourceId" -o tsv)

WS_GUID=$(az monitor log-analytics workspace show --ids "$WS_RESOURCE_ID" --query customerId -o tsv)

• WS_GUID → --set OmsAgent.workspaceID and az monitor log-analytics query -w
• WS_RESOURCE_ID → needed later to re-enable the addon
• DCR_ID → needed later to recreate the association

Resolving from the DCR (rather than guessing a cluster-named workspace) matters: clusters onboarded without an explicit workspace land in a regional default such as DefaultWorkspace-<subscription-id>-<region-code> in resource group DefaultResourceGroup-<region-code>, which bears no relation to the cluster name.

If the DCR does not exist yet

DCR_ID is empty when Container Insights was never enabled on the cluster. Bootstrap it by enabling the addon once — that provisions the DCR, the association, and a default workspace — then capture the values and disable it again before deploying:

if [ -z "$DCR_ID" ]; then
  # omit --workspace-resource-id to let AKS create/attach the regional default workspace,
  # or pass one explicitly if the user has a preferred workspace
  az aks enable-addons -a monitoring -g <rg> -n <name>

  DCR_ID=$(az monitor data-collection rule association list --resource "$AKS_RESOURCE_ID" \
    --query "[?name=='ContainerInsightsExtension'].dataCollectionRuleId | [0]" -o tsv)
  WS_RESOURCE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \
    --query "destinations.logAnalytics[0].workspaceResourceId" -o tsv)
  WS_GUID=$(az monitor log-analytics workspace show --ids "$WS_RESOURCE_ID" --query customerId -o tsv)
fi

Then continue into the normal flow (disable the addon, recreate the association, deploy). Record whether the addon was originally enabled: if it was never enabled, leave it disabled at cleanup rather than enabling something the user never had.

This bootstrap also creates the aad-msi-auth-token secret that the agent needs for MSI auth, so it is required even if you only care about the workspace values.

Determine the current production image

Do not hardcode a production tag — it goes stale. Read it from ReleaseNotes.md in the repo root, whose newest entry under ## Release History is the current production release:

PROD_TAG=$(grep -m1 -oP 'ciprod:\K[^ ]+(?= \(linux\))' ReleaseNotes.md)
PROD_TAG_WIN=$(grep -m1 -oP 'ciprod:\K[^ ]+(?= \(windows\))' ReleaseNotes.md)
echo "$PROD_TAG / $PROD_TAG_WIN"
# -> 3.8.0-ci-prod-09-03-2026-fd42f68c / win-3.8.0-ci-prod-09-03-2026-fd42f68c

grep -m1 takes the first match, and release notes are newest-first, so this always yields the latest release. The entries look like:

### 09/03/2026 -
##### Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.8.0-ci-prod-09-03-2026-fd42f68c (linux)
##### Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:win-3.8.0-ci-prod-09-03-2026-fd42f68c (windows)

These are already bare tags in the form this chart expects, and the Windows tag is read directly rather than derived — use it as-is.

Confirm your branch is not behind on release notes (git log -1 --format=%cd -- ReleaseNotes.md). If the value disagrees with what the cluster is actually running, prefer the live cluster image and tell the user:

kubectl get ds ama-logs -n kube-system \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs")]}{.image}{"\n"}{end}'

Generate the chart (mandatory first step)

charts/azuremonitor-containerinsights is template-only. It ships Chart-template.yaml and values-template.yaml; Chart.yaml and values.yaml are generated at build time and are gitignored (.gitignore:36-37), so generating them is safe and leaves the tree clean.

Running Helm before generating fails with Error: Chart.yaml file is missing.

Generate exactly as CI/CD does (.pipelines/helm-deploy-templates/ama-logs-helm-deploy.yaml):

cd charts/azuremonitor-containerinsights
export HELM_SEMVER="<linux-tag>"          # e.g. 3.1.34-17-g67321cf0d-20260323045331
export IMAGE_TAG="<linux-tag>"
export IMAGE_TAG_WINDOWS="<windows-tag>"  # e.g. win-3.1.34-17-g67321cf0d-20260323045331
envsubst '${HELM_SEMVER} ${IMAGE_TAG} ${IMAGE_TAG_WINDOWS}' < Chart-template.yaml > Chart.yaml
envsubst '${HELM_SEMVER} ${IMAGE_TAG} ${IMAGE_TAG_WINDOWS}' < values-template.yaml > values.yaml

envsubst only expands ${...}. The <your_cluster_id> / <your_workspace_id> style placeholders are not substituted and must be passed as --set overrides at deploy time (see below).

Image tags for this chart

The image is assembled as mcr.microsoft.com + OmsAgent.imageRepository + : + OmsAgent.imageTagLinux.

So the tag is the bare tag, and the repo (ciprod vs cidev) is a separate value:

⚠️ Do not use the cidev:<tag> form here. That was the convention for the older azuremonitor-containerinsights-for-prod-clusters chart. Passing cidev:3.1.34 as the tag renders …/ciprod:cidev:3.1.34, which is an invalid reference.

🚨 Silent-fallback footgun — this can invalidate an entire test run. If imageTagLinux / imageTagWindows are empty or unset, the chart does not fail. It silently falls back to the hardcoded defaults in get.addonImageTag (templates/_helpers.tpl): 3.1.34 and win-3.1.34. You would then be comparing production against production and conclude "no regression". Always verify the live image after deploying (see the verification step below).

Windows naming convention: prefix win- on the Linux tag — e.g. 3.8.0-ci-prod-09-03-2026-fd42f68c → win-3.8.0-ci-prod-09-03-2026-fd42f68c. For production tags read the Windows line from ReleaseNotes.md directly instead of deriving it; only derive for CI build tags, where the build log may list the Linux tag alone.

Deploy with Helm

Deploy with the same override set CI/CD uses. Always use --install so the command handles both fresh installs and upgrades:

AKS_RESOURCE_ID="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/<name>"

helm upgrade --install ama-logs charts/azuremonitor-containerinsights -n kube-system \
  --set global.commonGlobals.CloudEnvironment=azurepubliccloud \
  --set global.commonGlobals.Region=<region> \
  --set global.commonGlobals.Versions.Kubernetes=<k8s-version> \
  --set Azure.Cluster.ResourceId="$AKS_RESOURCE_ID" \
  --set OmsAgent.aksResourceID="$AKS_RESOURCE_ID" \
  --set OmsAgent.accessTokenSecretName=aad-msi-auth-token \
  --set OmsAgent.workspaceID="$WS_GUID" \
  --set OmsAgent.imageRepository=/azuremonitor/containerinsights/<ciprod|cidev> \
  --set OmsAgent.imageTagLinux=<linux-tag> \
  --set OmsAgent.imageTagWindows=<windows-tag> \
  --timeout 10m

Notes:

• accessTokenSecretName=aad-msi-auth-token is required on MSI-auth clusters. The chart default is ama-logs-secret (legacy workspace-key auth); leaving the default breaks token acquisition.
• CI/CD additionally passes --server-side=true --force-conflicts. Those are Helm 4 flags — the pipeline pins helmVersionToInstall: 'latest'. On Helm 3 they fail with Error: unknown flag: --server-side, so omit them (check with helm version --short).
• The namespace is hardcoded to kube-system inside the templates.
• The chart renders a Secret named ama-logs-secret, which is a different object from aad-msi-auth-token — installing does not clobber the MSI token secret.
• The chart only mounts container-azm-ms-agentconfig; it never creates it, so any scenario ConfigMap you apply for testing is safe from being overwritten.

Verify the deployment before collecting data

Because of the silent-fallback behaviour above, confirm the live image is the one you intended:

kubectl get ds ama-logs -n kube-system \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs")]}{.image}{"\n"}{end}'
kubectl get ds ama-logs-windows -n kube-system \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="ama-logs-windows")]}{.image}{"\n"}{end}'

Abort and fix the overrides if either does not match the tag under test.

Dry-run first (recommended)

Validate rendering without touching the cluster:

helm lint charts/azuremonitor-containerinsights --set global.commonGlobals.Versions.Kubernetes=<k8s-version>
helm template ama-logs charts/azuremonitor-containerinsights -n kube-system <same --set flags> \
  | grep -E '^\s+image: ' | sort -u

A correct render produces 9 objects — ClusterRole, ClusterRoleBinding, 2 ConfigMaps (container-azm-ms-aks-k8scluster, ama-logs-rs-config), 2 DaemonSets (Linux + Windows), 1 Deployment (replicaset), Secret, ServiceAccount.

AKS managed addon conflicts

If the target cluster has the managed Container Insights addon enabled, it owns the Helm release aks-managed-azure-monitor-logs and reconciles roughly every 20 minutes, reverting any backdoor install or kubectl set image.

Installing on top of it fails outright with an ownership error:

Error: Unable to continue with install: ServiceAccount "ama-logs" in namespace "kube-system" exists
and cannot be imported into the current release: invalid ownership metadata; annotation validation
error: key "meta.helm.sh/release-name" must equal "ama-logs": current value is
"aks-managed-azure-monitor-logs"

So disabling the addon is mandatory, not optional. You must already have DCR_ID and WS_RESOURCE_ID from "Resolve the workspace" — capture them before disabling, because the disable deletes the association:

az aks disable-addons -a monitoring -g <rg> -n <name> --yes   # --yes is required (no TTY prompt)

# recreate the association that the disable removed
az monitor data-collection rule association create --name ContainerInsightsExtension \
  --rule-id "$DCR_ID" --resource "$AKS_RESOURCE_ID"

The aad-msi-auth-token secret survives the disable, so MSI auth keeps working.

To restore afterwards — only if the addon was enabled when you started — uninstall your release first, then re-enable with the workspace the DCR pointed at:

helm uninstall ama-logs -n kube-system
az aks enable-addons -a monitoring -g <rg> -n <name> --workspace-resource-id "$WS_RESOURCE_ID"

Uninstalling first is required: the addon install hits the same ownership error in reverse if your release still owns ama-logs.

Check the exit code of az directly. Piping it into tail/grep masks failures behind the pipeline's exit status, so a wrong workspace name can look like success. Verify by re-reading addonProfiles.omsagent.enabled and the pod list rather than trusting the command's output.

Collect Table Data

Run Kusto queries via az monitor log-analytics query -w "$WS_GUID" (or the kusto-mcp MCP server if available).

Collect aggregated row counts in 1-minute bins from (deployment time + 5 min) to (deployment time + 10 min) for these tables:

• ContainerInventory
• KubeNodeInventory
• KubePodInventory
• InsightsMetrics
• Perf
• ContainerLogV2

Query template (run once per table, all 6 can run in parallel):

<TableName>
| where TimeGenerated between(datetime('<deployTime+5min>') .. datetime('<deployTime+10min>'))
| where _ResourceId =~ '<clusterResourceId>'
| summarize Count=count() by bin(TimeGenerated, 1m)
| order by TimeGenerated asc

Timing: Wait at least 15 minutes after deployment before running these queries — this accounts for pod startup (~5 min) plus Log Analytics ingestion latency (~5–10 min). The query window (deploy+5 to deploy+10) captures steady-state data only.

Compare Data Volume

1. Compare production vs test counts side by side for each table.
2. For ContainerInventory, KubeNodeInventory, KubePodInventory, InsightsMetrics, Perf: counts must match exactly per minute, excluding first/last minute edge windows. If they differ by even 1, investigate.
3. For ContainerLogV2: exact match is not required, but check for sustained upward/downward trends indicating regression.

Check Build Failure Reason

Query the build timeline to find which task(s) failed:

az devops invoke --organization "https://dev.azure.com/github-private" \
  --area build --resource timeline \
  --route-parameters project=microsoft buildId=<BUILD_ID> \
  --query "records[?result=='failed'].{name:name, type:type}" -o table

• If the only failed task name contains "Trivy" (vulnerability scan), the build images are valid — continue using this build. Do NOT fall back to a previous build. Extract the image tag from this build's logs.
• If any other task failed, the build is unusable — report the failure to the user.

Extract Image Version from Build Logs

Use the ADO API to read the build log directly (no need to download zip files):

1. Find the log ID for the "Multi-arch Linux build" task:

   az devops invoke --organization "https://dev.azure.com/github-private" \
     --area build --resource timeline \
     --route-parameters project=microsoft buildId=<BUILD_ID> \
     --query "records[?name=='Multi-arch Linux build'].{name:name, logId:log.id}" -o json
   

2. Read the log and extract the image tag. The log contains a line like:

   ##[warning]Linux image built with tag: containerinsightsprod.azurecr.io/public/azuremonitor/containerinsights/cidev:3.1.34-17-g67321cf0d-20260323045331
   

   Extract the bare tag only — everything after the cidev: — because this chart takes the repository and the tag as separate values:

   grep -o 'cidev:[^ ]*' <log> | head -1 | cut -d: -f2
   # -> 3.1.34-17-g67321cf0d-20260323045331
   

   The cidev part becomes OmsAgent.imageRepository=/azuremonitor/containerinsights/cidev.

3. Derive the Windows tag from the Linux tag using the naming convention (prefix win-). Alternatively, find "Docker windows build for multi-arc image" log for a line like:

   ##[warning]Windows image built with tag: ...cidev:win-3.1.34-17-g67321cf0d-20260323045331
   

Get PodUid

Query KubePodInventory scoped to the relevant deployment window:

KubePodInventory
| where TimeGenerated between(datetime('<windowStart>') .. datetime('<windowEnd>'))
| where _ResourceId =~ '<clusterResourceId>'
| where Name in ('<pod1>', '<pod2>', ...)
| distinct PodUid, Name

Compare Resource Consumption

Query per-minute resource consumption. You can batch multiple pods in one query using or:

Perf
| where TimeGenerated between(datetime('<windowStart>') .. datetime('<windowEnd>'))
| where _ResourceId =~ '<clusterResourceId>'
| where CounterName =~ '<counterName>'
| where InstanceName contains '<podUid1>' or InstanceName contains '<podUid2>' or ...
| extend Pod = case(
    InstanceName contains '<podUid1>', '<podName1>',
    InstanceName contains '<podUid2>', '<podName2>',
    'unknown')
| summarize MaxValue=max(CounterValue/1000/1000/1000) by bin(TimeGenerated, 1m), Pod
| order by Pod asc, TimeGenerated asc

Compare the two counter names:

• memoryWorkingSetBytes — memory in GB
• cpuUsageNanoCores — CPU in cores

Flag any regression (sustained increase in the test deployment).

Investigate Data Volume Regression

When a table's counts differ between production and test (or ContainerLogV2 shows a sustained trend), investigate before marking it as a regression:

1. Break down by ContainerName in both windows to identify which container(s) are responsible:

   <TableName>
   | where TimeGenerated between(datetime('<windowStart>') .. datetime('<windowEnd>'))
   | where _ResourceId =~ '<clusterResourceId>'
   | summarize Count=count() by ContainerName
   | sort by Count desc
   

2. Compare the per-container breakdown between production and test. Look for:

   • Containers present in one window but not the other (cluster workload change, not a code regression).
   • A specific container with significantly higher counts in the test window.

3. If a container is only present in one window, verify it was running independently of the deployment by checking a broader time range (e.g., 30 min before the deployment):

   <TableName>
   | where TimeGenerated between(datetime('<deployTime-30min>') .. datetime('<deployTime>'))
   | where _ResourceId =~ '<clusterResourceId>'
   | where ContainerName == '<suspectContainer>'
   | summarize Count=count() by bin(TimeGenerated, 1m)
   | order by TimeGenerated asc
   

4. Classify the finding:

   • If the difference is caused by a container that started/stopped independently of the deployment → not a regression (cluster workload difference). Note this in the output file and mark as PASS.
   • If the difference is caused by an ama-logs container or directly relates to the code change → potential regression. Flag it and ask the user to review.

Investigate Resource Consumption Regression

When memory or CPU shows a sustained increase in the test deployment:

1. Check per-container resource usage within each pod to isolate which container is consuming more. The ama-logs pods run multiple containers (ama-logs, ama-logs-prometheus, addon-token-adapter). Use:

   Perf
   | where TimeGenerated between(datetime('<windowStart>') .. datetime('<windowEnd>'))
   | where _ResourceId =~ '<clusterResourceId>'
   | where CounterName =~ '<counterName>'
   | where InstanceName contains '<podUid>'
   | summarize MaxValue=max(CounterValue/1000/1000/1000) by bin(TimeGenerated, 1m), InstanceName
   | order by InstanceName asc, TimeGenerated asc
   

2. Compare the per-container breakdown between production and test to pinpoint the specific container causing the increase.

3. Classify the finding:

   • Increases < 10% within normal variance → not a regression. Note in output and mark as PASS.
   • Sustained increases ≥ 10% in an ama-logs container → potential regression. Flag and ask user to review.

Steps

The workflow has two parallel tracks that converge after the build completes.

Phase 1: Obtain Build + Deploy Production Image (parallel)

1. Collect inputs and derive values (see Required Inputs and Derived Values tables). Only the branch name and cluster resource ID come from the user. Resolve the production image from ReleaseNotes.md (see "Determine the current production image") and the workspace from the DCR (see "Resolve the workspace", including the bootstrap path if no DCR exists yet). Save all values to the output file.
2. Set kubectl context: kubectl config use-context <cluster name>. Then record whether the AKS managed monitoring addon is enabled — cleanup must put it back the way you found it:

   az aks show -g <rg> -n <name> --query "addonProfiles.omsagent.enabled" -o tsv
   

   If true, it will revert your backdoor deployment every ~20 minutes — disable it and recreate the DCR association (see "AKS managed addon conflicts").
3. Check for an existing build on the branch for the latest commit (definition ID 444, org: github-private, project: microsoft).
   • If a completed build exists on the latest commit → use it (even if it failed due to Trivy — see "Check Build Failure Reason").
   • IMPORTANT: A build that failed ONLY due to Trivy is still usable. Do NOT fall back to a previous build. The images are already built and pushed before Trivy runs. Always extract the image tag from the failed build's logs (see "Extract Image Version from Build Logs").
   • If no usable build exists → trigger a new build. Save the build ID.
4. If the build is already complete, skip to Phase 2 after finishing production baseline steps. If the build is still running, proceed with steps 5–9 in parallel; periodically check build status during wait times.
5. Generate the chart and deploy the current production image (see "Generate the chart", "Image tags for this chart", and "Deploy with Helm") using imageRepository=/azuremonitor/containerinsights/ciprod. Verify the live image matches the prod tag before continuing (see "Verify the deployment before collecting data"). Record the production deployment time (UTC).
6. Wait 15 minutes, then verify pods: kubectl get pods -n kube-system | grep ama-logs. Confirm all are Running with 0 restarts. Save pod names to the output file.
7. Collect production baseline data for all 6 tables (see "Collect Table Data"). Save results to the output file.

Phase 2: Deploy Test Image (after build completes)

8. Confirm the build completed. Check failure reason if needed (see "Check Build Failure Reason"). If it failed for a non-Trivy reason, ask the user whether to retrigger. If it failed only due to Trivy, treat it as a successful build — the images are valid. Do NOT fall back to a previous build.
9. Extract the test image version from the build logs (see "Extract Image Version from Build Logs"). Save to the output file.
10. Regenerate the chart with the test tags and deploy using imageRepository=/azuremonitor/containerinsights/cidev and the bare tags from step 9. Verify the live image matches the test tag before continuing — the chart silently falls back to 3.1.34 if a tag is empty, which would make the comparison meaningless. Record the test deployment time (UTC).
11. Wait 15 minutes, then verify pods are Running. If any pod restarted, get the reason via kubectl describe pod <name> -n kube-system. Save pod names to the output file.
12. Collect test data for all 6 tables (see "Collect Table Data"). Save results to the output file.

Phase 3: Compare Results

13. Compare data volume between production and test for all tables (see "Compare Data Volume"). If any table shows a difference, investigate before reporting (see "Investigate Data Volume Regression").
14. Get PodUid for all pods in both deployments (see "Get PodUid").
15. Compare resource consumption for memoryWorkingSetBytes and cpuUsageNanoCores (see "Compare Resource Consumption"). If any metric shows a sustained increase, investigate before reporting (see "Investigate Resource Consumption Regression").
16. Restore the cluster: helm uninstall ama-logs -n kube-system, then put the addon back the way you found it (step 2) — re-enable it with $WS_RESOURCE_ID if it was originally enabled, or leave it disabled if it never was. Confirm 5/5 ama-logs pods are Running on the prod image and that both ContainerInsightsExtension and ContainerInsightsMetricsExtension DCR associations exist. Delete any test ConfigMaps you applied.
17. Write summary to the output file: pass/fail for each table and resource check. Include the exact prod and test images as read back from the live DaemonSets, not just the intended tags. Include investigation findings for any anomalies — clearly distinguish between code regressions and cluster workload differences.