tools

作者: astronomer

在编写、编辑或审查此仓库中的命令行工具和辅助脚本(位于bin/目录下)时使用。涵盖工具的存放位置、参数解析等内容……

npx skills add https://github.com/astronomer/astronomer --skill tools

Writing tools in bin/

Repo tooling (setup scripts, one-off utilities, anything a human or CI invokes directly) lives in bin/ and follows a few rules so tools are discoverable, self-documenting, and safe to run by accident.


Critical Rules

  1. Tools live in bin/ as executable scripts: a shebang (#!/usr/bin/env python3 for Python) plus chmod +x. Python tools run via uv run bin/<tool>.py.
  2. Every tool parses arguments with argparse (or the language equivalent) so --help works and every argument is self-documenting. Parse arguments as the first thing main() does.
  3. --help and insufficient/invalid arguments must do no work. They print usage and exit before any side effect. argparse gives this for free as long as parsing happens before any side-effecting code.
  4. A tool must not perform a destructive or state-mutating operation by default. Merely running it (or running it to read --help) must not create/delete Kubernetes objects, write/delete files, call external services, or change the active context.

Non-destructive by default

The failure mode to design against: someone runs bin/some-tool.py (or bin/some-tool.py --help) expecting it to be inert or to print help, and instead it mutates whatever ambient context it finds — the current kube context, the current directory, a live cluster.

The rule that prevents it: do not give a safe-looking default to any argument that determines where a mutation lands (a namespace, a cluster, a path, a target host). Make those arguments required with no default, so a bare or accidental invocation aborts before doing anything.

With argparse, a required=True argument with no default means:

  • tool (no args) → prints usage to stderr and exits non-zero, before main() reaches any side effect.
  • tool --help → prints help and exits 0.
  • tool --namespace foo ... → runs, because the caller was explicit about the target.

Worked example: bin/setup-forgejo-ca.py

This script creates and deletes Kubernetes Secrets in a cluster. It originally defaulted its namespaces (astronomer, git-forgejo). Running bin/setup-forgejo-ca.py --help to read the help text would instead have run the whole thing against the reader's current kube context — a potentially destructive surprise.

The fix was to make the namespaces required, with no defaults:

def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--platform-namespace", required=True, help="...")
    parser.add_argument("--forgejo-namespace", required=True, help="...")
    return parser.parse_args()

def main() -> None:
    args = parse_args()          # aborts here on a bare run or --help, before any kubectl
    ...                          # cluster mutations only happen after this line

Now --help and a bare run both abort before touching the cluster, and any real run has to name its target namespaces on purpose.

Automation still works

Making the target arguments required does not break automated callers — it just moves the intent to the caller, where it belongs. The automated invocation passes the values explicitly. For example, the git-sync-private-ca scenario's pre_helm_scripts entry names the namespaces:

pre_helm_scripts:
  - bin/setup-forgejo-ca.py --platform-namespace astronomer --forgejo-namespace git-forgejo

Checklist for a new or edited tool

  • Lives in bin/, is executable, has the right shebang.
  • Uses argparse; --help works and does nothing else.
  • Arguments that decide where a mutation lands are required with no default.
  • Side effects run only after arguments parse successfully.
  • Fails loudly on error (non-zero exit), and is idempotent (safe to re-run) where practical.
  • Callers (CI, scenario manifests, other scripts) pass the required arguments explicitly.

来自 astronomer 的更多技能

airflow
astronomer
查询、管理和排查Apache Airflow的DAG、运行记录、任务及系统配置。支持30多种命令,涵盖DAG检查、运行管理、任务日志、配置查询及直接REST API访问。通过持久化配置管理多个Airflow实例;自动发现本地和Astro部署。同步(等待完成)或异步触发DAG运行,诊断故障,清除运行记录以重试,并通过重试/映射索引过滤访问任务日志。输出...
official
airflow-hitl
astronomer
在Airflow DAG中使用可延迟操作符实现人工审批关卡、表单输入和分支。四种操作符类型:用于批准/拒绝决策的ApprovalOperator、带表单的多选项选择HITLOperator、人工驱动的任务路由HITLBranchOperator,以及表单数据收集HITLEntryOperator。所有操作符均为可延迟设计,在通过Airflow UI的"必需操作"标签页或REST API等待人工响应时释放工作槽位。支持包括自定义在内的可选功能...
official
airflow-state-store
astronomer
Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (`task_state_store`, `asset_state_store`) and the…
official
analyzing-data
astronomer
查询数据仓库,利用缓存的模式和概念映射来回答业务问题。支持对重复问题类型进行模式查找和缓存,并通过记录结果来改进后续查询。包含概念到表的映射缓存,以及通过INFORMATION_SCHEMA或代码库grep进行表结构发现。提供run_sql()和run_sql_pandas()内核函数,返回Polars或Pandas DataFrame用于分析。提供CLI命令用于管理概念、模式和表缓存,以及...
official
annotating-task-lineage
astronomer
使用入口和出口为Airflow任务标注数据血缘。支持使用OpenLineage Dataset对象、Airflow Assets和Airflow Datasets定义跨数据库、数据仓库及云存储的输入输出。当运算符缺少内置OpenLineage提取器时作为备用方案;遵循四级优先级系统,其中自定义提取器和OpenLineage方法优先。包含针对Snowflake、BigQuery、S3和PostgreSQL的数据集命名辅助工具,以确保一致性...
official
authoring-dags
astronomer
创建Apache Airflow DAG的引导式工作流,集成验证与测试。采用六阶段结构化方法:发现环境与现有模式、规划DAG结构、遵循最佳实践实现、通过af CLI命令验证、经用户同意测试、迭代修复。用于发现(af config connections、af config providers、af dags list)和验证(af dags errors、af dags get、af dags explore)的CLI命令可提供DAG的即时反馈...
official
authoring-go-sdk-tasks
astronomer
Writes Airflow task logic in Go using the Airflow Go SDK. Use when the user wants to implement Airflow tasks in Go, asks about `BundleProvider`/`RegisterDags`,…
official
authoring-java-sdk-tasks
astronomer
使用 Airflow Java SDK 编写 Java、Kotlin 或任何 JVM 语言的 Airflow 任务逻辑。当用户想要在 Java/JVM 中实现 Airflow 任务时使用,询问……
official