mcore-cicd

CI/CD Guide

Answer-First CI Facts

For PR-label or trigger questions, lead with the exact values:

No label: scope=mr-github-slim, n_repeat=2, lightweight=false.
container::lts only switches the container image path to LTS and combines with any scope label. Opt-in only — attach it solely when the user explicitly asks for LTS validation; never add it on your own initiative, even for a container or dependency change.
Run MBridge tests additionally triggers the MBridge L1 suite.
Run NeMoRL tests additionally triggers NeMo RL's Megatron functional test suite.
⚠️ WARNING — destructive remote write. tools/trigger_internal_ci.py force-pushes the current branch to the internal GitLab remote as pull-request/<branch>. Always run with --dry-run first and confirm the destination ref before invoking it without the flag. Never run against a shared or protected branch — only target your own pull-request branch. Safe preflight: python tools/trigger_internal_ci.py --gitlab-origin gitlab --dry-run. Add the optional --functional-test-* flags only after the dry-run output matches the intended destination.

CI Pipeline Structure

The main workflow is .github/workflows/cicd-main.yml. It triggers on pushes to branches matching pull-request/[0-9]+ and deploy-release/*, on merge groups, and on manual dispatch.

is-not-external-contributor
  └─ pre-flight
       └─ configure          # determines scope, container tag, n_repeat
            ├─ linting
            ├─ cicd-container-build
            │    ├─ cicd-parse-unit-tests → cicd-unit-tests-latest
            │    ├─ cicd-parse-integration-tests-h100 → cicd-integration-tests-latest-h100
            │    └─ cicd-parse-integration-tests-gb200 → cicd-integration-tests-latest-gb200 (maintainers only)
            └─ Nemo_CICD_Test  # final pass/fail gate

Images are pushed to:

AWS ECR: 766267172432.dkr.ecr.us-east-1.amazonaws.com/…
GCP Artifact Registry: us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-lm/…

CI Test Scope Labels

The CI pipeline reads PR labels to decide test scope, n_repeat, and container image.

Decision tree (first match wins):

Condition	`scope`	`n_repeat`	`lightweight`	Notes
Merge group	`mr-github`	1	false	Automatic, no label needed
(no label)	`mr-github-slim`	2	false	Slim subset only

Orthogonal image label:

Label	Effect
`container::lts`	Build on the older long-term-support NGC PyTorch base instead of `dev`'s latest — a backward-compat check, not a different test set (combinable with any scope label)
`Run MBridge tests`	Also triggers the MBridge L1 test suite
`Run NeMoRL tests`	Also triggers NeMo RL's Megatron functional test suite

Adding a label does not itself start cicd-main.yml; apply it before the next synthetic PR push or rerun the workflow after applying it.

Which label to attach when opening a PR

Changed paths / nature of change	Label to attach
Docs only (`docs/`, `*.md`, docstrings)	(none)
CI/tooling only (`.github/`, `tools/`, `Makefile`)	(none)
Touches MBridge integration	add `Run MBridge tests`
Could affect NeMo RL's Megatron integration	add `Run NeMoRL tests`

Triggering Internal CI

Use tools/trigger_internal_ci.py after the internal GitLab remote and GITLAB_TOKEN are configured; see @tools/trigger_internal_ci.md for setup details. First run a dry run and verify the destination ref:

python tools/trigger_internal_ci.py --gitlab-origin gitlab --dry-run

The script force-pushes the current branch to pull-request/<branch> before triggering the pipeline. Only target your own pull-request branch, never a shared or protected branch. Add optional --functional-test-* flags only after the dry-run output matches the intended destination.

CI Failure Investigation

CI branches always follow the pattern pull-request/<number>.

Locating the PR from a CI Branch

# Extract PR number from the current branch
PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')

# Fetch the PR metadata (title, labels, author, base branch)
gh pr view "$PR_NUMBER" --repo NVIDIA/Megatron-LM

# Show the changeset for that PR
gh pr diff "$PR_NUMBER" --repo NVIDIA/Megatron-LM

Reading CI Job Logs

# List recent workflow runs for the PR
gh run list --repo NVIDIA/Megatron-LM --branch "pull-request/$PR_NUMBER"

# Stream failing job output
gh run view <run-id> --repo NVIDIA/Megatron-LM --log-failed

Full per-rank logs are not in the runner stdout. They are uploaded as GitHub artifacts named logs-<test_case>-<run_id>-<uuid>.

# 1. Find artifact name
gh run view <run-id> --repo NVIDIA/Megatron-LM --json artifacts \
  --jq '.artifacts[].name'

# 2. Download the artifact zip
gh run download <run-id> --repo NVIDIA/Megatron-LM \
  --name "logs-<artifact-name>" -D ./ci-logs

# 3. Locate which rank logs contain errors
grep -r -l "ERROR\|Traceback\|FAILED\|fatal" ./ci-logs/

# 4. Log files can exceed 10 000 lines — never read a full log at once.
wc -l ./ci-logs/<test>/<attempt>/attempt_0/<rank>/stderr.log
sed -n '1,200p' ./ci-logs/.../stderr.log   # read in chunks

Identifying Failure Root Cause

Linting failure — re-run tools/autoformat.sh locally; the diff shows exactly what needs to change.
Container build failure — inspect the cicd-container-build job log.
Unit test failure — the failing bucket is in the cicd-unit-tests-latest job matrix.
Functional test failure — look at the cicd-integration-tests-* job. Start with stdout.log for rank 0.
Flaky test — the runner retries automatically up to 3 times. If all retries exhausted and the pattern matches a known transient (NCCL, ECC, segfault), it is infrastructure noise.

Correlating a Failure with the PR Changeset

# Find unit tests that cover a changed source file
grep -r "from megatron.core.transformer.attention" tests/unit_tests/ -l

# Check CODEOWNERS for reviewer assignment
cat .github/CODEOWNERS | grep "<changed-path>"