discovering-data

Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (`task_state_store`, `asset_state_store`) and the…

Custom OpenLineage extractors for unsupported Airflow operators and complex lineage scenarios. Two approaches: add OpenLineage methods directly to operators you own (recommended), or create custom extractors for third-party operators you cannot modify Extractors intercept operator execution at three points: before execution for static lineage, after success for runtime-determined outputs, and optionally after failure for partial lineage Register extractors via airflow.cfg or environment...

Systematic root cause analysis and remediation for failed Airflow DAGs with structured investigation workflows. Guides through four-step diagnosis process: identify the failure, extract error details, gather contextual information, and deliver actionable remediation steps Categorizes failures into four types (data, code, infrastructure, dependency) to focus investigation and suggest appropriate fixes Provides ready-to-use CLI commands for log retrieval, run comparison, task clearing, and DAG...

Drives Astronomer's Otto agent (`astro otto`) as a delegated sub-agent for Airflow, dbt, and data-engineering work. Use when the user explicitly asks to "use…

Deploy Airflow DAGs and projects. Use when the user wants to deploy code, push DAGs, set up CI/CD, deploy to production, or asks about deployment strategies…

Builds, packs, and deploys compiled Airflow Go SDK bundles so the ExecutableCoordinator can run them. Use when the user wants to compile a Go task bundle, asks…

Iterative test-debug-fix cycles for Airflow DAGs with comprehensive failure diagnosis. Start with af runs trigger-wait <dag_id> to run a DAG and wait for completion; no pre-flight checks needed On failure, use af runs diagnose for comprehensive failure summary and af tasks logs to inspect error details from specific tasks Supports custom configuration, timeouts, and retry attempts; handles success, failure, and timeout scenarios with clear response interpretation Quick validation available...

Trace downstream data lineage to assess change impact before modifying tables or DAGs. Identifies direct consumers of a target table or DAG through source code search, view dependencies, and BI tool connections Builds a full dependency tree mapping all downstream impacts, from tables to dashboards to ML models Categorizes dependencies by criticality (critical, high, medium, low) to prioritize stakeholder communication and testing Generates an impact report with risk assessment, affected...