debugging-dags작성자: astronomer
Systematic root cause analysis and remediation for failed Airflow DAGs with structured investigation workflows. Guides through four-step diagnosis process: identify the failure, extract error details, gather contextual information, and deliver actionable remediation steps Categorizes failures into four types (data, code, infrastructure, dependency) to focus investigation and suggest appropriate fixes Provides ready-to-use CLI commands for log retrieval, run comparison, task clearing, and DAG...
npx skills add https://github.com/astronomer/agents --skill debugging-dagsDAG Diagnosis
You are a data engineer debugging a failed Airflow DAG. Follow this systematic approach to identify the root cause and provide actionable remediation.
Running the CLI
These commands assume af is on PATH. Run via astro otto to get it automatically, or install standalone with uv tool install astro-airflow-mcp.
Step 1: Identify the Failure
If a specific DAG was mentioned:
- Run
af runs diagnose <dag_id> <dag_run_id>(if run_id is provided) - If no run_id specified, run
af dags statsto find recent failures
If no DAG was specified:
- Run
af healthto find recent failures across all DAGs - Check for import errors with
af dags errors - Show DAGs with recent failures
- Ask which DAG to investigate further
Step 2: Get the Error Details
Once you have identified a failed task:
- Get task logs using
af tasks logs <dag_id> <dag_run_id> <task_id> - Look for the actual exception - scroll past the Airflow boilerplate to find the real error
- Categorize the failure type:
- Data issue: Missing data, schema change, null values, constraint violation
- Code issue: Bug, syntax error, import failure, type error
- Infrastructure issue: Connection timeout, resource exhaustion, permission denied
- Dependency issue: Upstream failure, external API down, rate limiting
Step 3: Check Context
Gather additional context to understand WHY this happened:
- Recent changes: Was there a code deploy? Check git history if available
- Data volume: Did data volume spike? Run a quick count on source tables
- Upstream health: Did upstream tasks succeed but produce unexpected data?
- Historical pattern: Is this a recurring failure? Check if same task failed before
- Timing: Did this fail at an unusual time? (resource contention, maintenance windows)
Use af runs get <dag_id> <dag_run_id> to compare the failed run against recent successful runs.
On Astro
If you're running on Astro, these additional tools can help with diagnosis:
- Deployment activity log: Check the Astro UI for recent deploys — a failed deploy or recent code change is often the cause of sudden failures
- Astro alerts: Configure alerts in the Astro UI for proactive failure monitoring (DAG failure, task duration, SLA miss)
- Observability: Use the Astro observability dashboard to track DAG health trends and spot recurring issues
On OSS Airflow
- Airflow UI: Use the DAGs page, Graph view, and task logs to inspect recent runs and failures
Step 4: Provide Actionable Output
Structure your diagnosis as:
Root Cause
What actually broke? Be specific - not "the task failed" but "the task failed because column X was null in 15% of rows when the code expected 0%".
Impact Assessment
- What data is affected? Which tables didn't get updated?
- What downstream processes are blocked?
- Is this blocking production dashboards or reports?
Immediate Fix
Specific steps to resolve RIGHT NOW:
- If it's a data issue: SQL to fix or skip bad records
- If it's a code issue: The exact code change needed
- If it's infra: Who to contact or what to restart
Prevention
How to prevent this from happening again:
- Add data quality checks?
- Add better error handling?
- Add alerting for edge cases?
- Update documentation?
Quick Commands
Provide ready-to-use commands:
- To clear and rerun the entire DAG run:
af runs clear <dag_id> <run_id> - To clear and rerun specific failed tasks:
af tasks clear <dag_id> <run_id> <task_ids> -D - To delete a stuck or unwanted run:
af runs delete <dag_id> <run_id>
astronomer의 다른 스킬
airflow
by astronomer
Query, manage, and troubleshoot Apache Airflow DAGs, runs, tasks, and system configuration. Supports 30+ commands across DAG inspection, run management, task logging, configuration queries, and direct REST API access Manage multiple Airflow instances with persistent configuration; auto-discover local and Astro deployments Trigger DAG runs synchronously (wait for completion) or asynchronously, diagnose failures, clear runs for retry, and access task logs with retry/map-index filtering Output...
airflow-hitl
by astronomer
Human approval gates, form inputs, and branching in Airflow DAGs using deferrable operators. Four operator types: ApprovalOperator for approve/reject decisions, HITLOperator for multi-option selection with forms, HITLBranchOperator for human-driven task routing, and HITLEntryOperator for form data collection All operators are deferrable, releasing worker slots while awaiting human response via Airflow UI's Required Actions tab or REST API Supports optional features including custom...
airflow-plugins
by astronomer
Build Airflow 3.1+ plugins that embed FastAPI apps, custom UI pages, React components, middleware, macros, and operator links directly into the Airflow UI. Use…
analyzing-data
by astronomer
Query your data warehouse to answer business questions with cached patterns and concept mappings. Supports pattern lookup and caching for repeated question types, with outcome recording to improve future queries Includes concept-to-table mapping cache and table schema discovery via INFORMATION_SCHEMA or codebase grep Provides run_sql() and run_sql_pandas() kernel functions returning Polars or Pandas DataFrames for analysis CLI commands for managing concept, pattern, and table caches, plus...
annotating-task-lineage
by astronomer
Annotate Airflow tasks with data lineage using inlets and outlets. Supports OpenLineage Dataset objects, Airflow Assets, and Airflow Datasets for defining inputs and outputs across databases, data warehouses, and cloud storage Use as a fallback when operators lack built-in OpenLineage extractors; follows a four-tier precedence system where custom extractors and OpenLineage methods take priority Includes dataset naming helpers for Snowflake, BigQuery, S3, and PostgreSQL to ensure consistent...
authoring-dags
by astronomer
Guided workflow for creating Apache Airflow DAGs with validation and testing integration. Structured six-phase approach: discover environment and existing patterns, plan DAG structure, implement following best practices, validate with af CLI commands, test with user consent, and iterate on fixes CLI commands for discovery ( af config connections , af config providers , af dags list ) and validation ( af dags errors , af dags get , af dags explore ) provide immediate feedback on DAG...
blueprint
by astronomer
Define reusable Airflow task group templates with Pydantic validation and compose DAGs from YAML. Use when creating blueprint templates, composing DAGs from…
checking-freshness
by astronomer
Verify data freshness by checking table timestamps and update patterns against a staleness scale. Identifies timestamp columns using common ETL naming patterns ( _loaded_at , _updated_at , created_at , etc.) and queries their maximum values to determine age Classifies data into four freshness statuses: Fresh (< 4 hours), Stale (4–24 hours), Very Stale (> 24 hours), or Unknown (no timestamp found) Provides SQL templates for checking last update time and row count trends over recent days to...