triage-leaked-infra

Triage Leaked Infrastructure

Assess whether an AWS infrastructure set (VPC + associated resources) from HyperShift CI is safe to delete. Every claim must be backed by an empirical AWS query — never assume, always verify.

Input

The user provides one of:

• cleanleaked tool output (an infra set block)
• A VPC ID (e.g., vpc-0abc123...)
• An infraID (e.g., 00ab3695c5f73d4354b9 or node-pool-78vcg)

Extract the infraID and VPC ID from the input. If only one is given, derive the other:

• VPC ID → aws ec2 describe-vpcs --vpc-ids <VPC> --query 'Vpcs[0].Tags' → extract infraID from kubernetes.io/cluster/<infraID> tag or Name tag (strip -vpc suffix)
• infraID → aws ec2 describe-vpcs --filters "Name=tag:Name,Values=<infraID>-vpc" → get VPC ID

Checks

Run these in order. For each, report PASS (safe signal), FAIL (do NOT delete), or UNKNOWN (could not determine). Use --region us-east-1 for all commands.

1. Protection tags

aws ec2 describe-vpcs --vpc-ids <VPC> --query 'Vpcs[0].Tags' --output json

• Check for hypershift.openshift.io/do-not-delete=true → if present: FAIL
• Check for hypershift.openshift.io/ci-cluster → if present: FAIL (this is a management cluster)

2. Protected VPC name

From the Name tag: if it is hypershift-ci-2-vpc, hypershift-ci-3-vpc, or hypershift-ci-metrics-vpc → FAIL

3. Protected user

Check if the infraID contains any of these usernames: aabdelre, agarcial, ahmed, alamela, alesross, bclement, brcox, celebdor, cewong, dario, dari2o, dmace, glipceanu, jiezhao, jparrill, mbhalodi, mbrown, meha, mgencur, mulham, mraee, rkshirsa, sdminonne, sjenning, tsegura, vismishr

If match → FAIL (developer cluster, contact the owner)

4. Expiration date / Age

From the VPC tags, check expirationDate:

• If present and not yet expired → FAIL
• If present and expired → PASS
• If absent → check resource age instead (the expirationDate tag only exists on resources created in the last ~2 weeks)

When expirationDate is absent, determine age from the earliest timestamped sub-resource:

aws ec2 describe-vpc-endpoints --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'VpcEndpoints[*].CreationTimestamp' --output text
aws ec2 describe-network-interfaces --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'NetworkInterfaces[*].Attachment.AttachTime' --output text

Take the earliest timestamp found. If the resource is older than 24 hours AND the CI pattern check (step 5) is PASS → PASS. If younger than 24 hours → FAIL. If no timestamp found at all → UNKNOWN.

5. CI pattern match

Does the infraID match a known CI test pattern?

• Hex ID (20+ lowercase hex chars, e.g., 00ab3695c5f73d4354b9) → CI generic e2e → PASS
• Named test: create-cluster-*, node-pool-*, control-plane-upgrade-*, autoscaling-*, karpenter-*, karpenter-upgrade-control-plane-*, scale-from-zero-*, kms-verify-*, request-serving-*, private-*, proxy-*, spot-demo-*, ha-break-glass-creds-*, custom-config-*, ho-upgrade-*, multi-hop-upgrade-* → PASS
• None of the above (e.g., hc1-*, clust-*, dev-*, test-dev-*) → UNKNOWN (might be a developer cluster)

6. OIDC S3 liveness

aws s3api head-object --bucket hypershift-ci-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1
aws s3api head-object --bucket hypershift-ci-2-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1
aws s3api head-object --bucket hypershift-ci-3-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1

If any returns 200 → FAIL (cluster still active). If all return 404/error → PASS.

7. EC2 instances

aws ec2 describe-instances --region us-east-1 \
  --filters "Name=tag:kubernetes.io/cluster/<infraID>,Values=owned" \
           "Name=instance-state-name,Values=pending,running,stopping,stopped" \
  --query 'Reservations[*].Instances[*].[InstanceId,State.Name,Tags[?Key==`Name`].Value|[0]]' --output text

If any instances returned → FAIL. If none → PASS.

8. red-hat-managed check

aws ec2 describe-instances --region us-east-1 \
  --filters "Name=vpc-id,Values=<VPC>" "Name=tag:red-hat-managed,Values=true" \
  --query 'Reservations[*].Instances[*].InstanceId' --output text

If any instances returned → FAIL (ROSA managed infrastructure). If none → PASS.

9. Sub-resources inventory (VPC-scoped)

Query each and report ID + Name for every resource, not just counts:

aws elbv2 describe-load-balancers --region us-east-1 --query "LoadBalancers[?VpcId=='<VPC>'].[LoadBalancerName,Type,Scheme,DNSName]" --output text
aws ec2 describe-vpc-endpoints --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'VpcEndpoints[*].[VpcEndpointId,VpcEndpointType,ServiceName,State]' --output text
aws ec2 describe-vpc-endpoint-service-configurations --region us-east-1 --output json | # filter by kubernetes.io/cluster/<infraID> tag
aws ec2 describe-nat-gateways --region us-east-1 --filter "Name=vpc-id,Values=<VPC>" --query 'NatGateways[*].[NatGatewayId,State,Tags[?Key==`Name`].Value|[0],NatGatewayAddresses[0].PublicIp]' --output text
aws ec2 describe-internet-gateways --region us-east-1 --filters "Name=attachment.vpc-id,Values=<VPC>" --query 'InternetGateways[*].[InternetGatewayId,Tags[?Key==`Name`].Value|[0]]' --output text
aws ec2 describe-subnets --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'Subnets[*].[SubnetId,CidrBlock,AvailabilityZone,Tags[?Key==`Name`].Value|[0]]' --output text
aws ec2 describe-security-groups --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'SecurityGroups[*].[GroupId,GroupName]' --output text
aws ec2 describe-network-interfaces --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'NetworkInterfaces[*].[NetworkInterfaceId,InterfaceType,Description]' --output text
aws ec2 describe-route-tables --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'RouteTables[*].[RouteTableId,Associations[0].Main,Tags[?Key==`Name`].Value|[0]]' --output text
aws ec2 describe-addresses --region us-east-1 --filters "Name=domain,Values=vpc" --output text | # cross-check with NAT gateway EIPs

This check is informational — always PASS but list every resource with its ID and name.

10. Route53 zones

aws route53 list-hosted-zones --output text --query 'HostedZones[*].[Id,Name,Config.PrivateZone,ResourceRecordSetCount]'

Search for zones matching <infraID>.ci.hypershift.devcluster.openshift.com and <infraID>.hypershift.local. For each found zone, list its records:

aws route53 list-resource-record-sets --hosted-zone-id <ZONE_ID> --query 'ResourceRecordSets[*].[Name,Type]' --output text

Report zone IDs, names, private/public, and record count. Informational — always PASS.

11. OIDC IAM provider

aws iam list-open-id-connect-providers --output json

Search for providers whose ARN contains the infraID (pattern: oidc-provider/hypershift-ci-*-oidc.s3.*.amazonaws.com/<infraID>). Report if found — this is an orphaned IAM resource that should be cleaned up with the infra set.

12. IAM roles (by infraID prefix)

aws iam list-roles --query "Roles[?starts_with(RoleName, '<infraID>')].[RoleName,CreateDate]" --output text

Report any IAM roles whose name starts with the infraID. These are orphaned OIDC-type roles (cloud-controller, ebs-csi, ingress, etc.) that belong to this infra set.

Output

Present the report in this format:

## Triage: <infraID>

| # | Check | Result | Detail |
|---|-------|--------|--------|
| 1 | Protection tags | PASS | No do-not-delete or ci-cluster tag |
| 2 | Protected VPC name | PASS | Name is "abcdef1234-vpc" |
| 3 | Protected user | PASS | No username match |
| 4 | Expiration date | PASS | Expired 2026-07-03 (5 days ago) |
| 5 | CI pattern match | PASS | Hex infraID (e2e-generic) |
| 6 | OIDC S3 liveness | PASS | Not found in any bucket |
| 7 | EC2 instances | PASS | 0 instances |
| 8 | red-hat-managed | PASS | No managed instances |
| 9 | Sub-resources | INFO | 1 IGW, 1 subnet, 1 RTB |
| 10 | Route53 zones | INFO | 2 zones found |
| 11 | OIDC IAM provider | INFO | 1 orphaned provider found |
| 12 | IAM roles | INFO | 0 orphaned roles |

### Verdict: SAFE TO DELETE
This infra set has no protection tags, no running instances, no OIDC document,
and the infraID matches a CI hex pattern. All deletion checks pass.

Verdict rules

These are non-negotiable:

• Any FAIL → verdict is DO NOT DELETE (explain which check failed and why)
• All checks PASS and CI pattern matches → SAFE TO DELETE
• All checks PASS but CI pattern is UNKNOWN → UNCERTAIN (could be a developer cluster)
• If CI pattern is PASS and age/expiration is PASS (older than 24h or expired), remaining UNKNOWN checks do not block → SAFE TO DELETE (a confirmed CI resource older than 24h with no OIDC, no instances, and no protection tags is leaked)
• If CI pattern is UNKNOWN and ANY check is UNKNOWN → UNCERTAIN — REQUIRES HUMAN DECISION