configuring-airflow-language-sdks

Configuring Airflow for Language SDKs

To run language SDK tasks, Airflow needs to know two things: which coordinator launches the native subprocess, and which queue routes to that coordinator. The mechanism is identical across every language SDK — only each coordinator's classpath and kwargs differ. This skill documents the shared wiring once, then the per-coordinator options. It is platform-neutral: the same settings apply on open-source Airflow and on managed platforms like Astro.

Experimental. The language SDKs are in preview; configuration keys may change.

For the task code, see authoring-language-sdk-tasks (and the per-language authoring skill, e.g. authoring-go-sdk-tasks). For building and shipping the artifact, see the per-language deploy skill (e.g. deploying-go-sdk-bundles).

Prerequisites on the worker

• The runtime or artifact the SDK needs must be present on the worker nodes, because the coordinator spawns a native subprocess per task instance. The exact requirement is per-SDK — see Per-coordinator options (the Go SDK needs no language runtime: the bundle is a self-contained native executable, but it must be built for the worker's OS/arch).
• The compiled/native artifact(s) must be reachable on the worker. See the per-language deploy skill.
• The coordinators ship with the Airflow Task SDK (apache-airflow-task-sdk, installed with Airflow). No extra Python package is required.

The two settings

Both live in the [sdk] configuration section and apply to every language SDK:

1. coordinators — a JSON object mapping a coordinator name you choose to its implementation (classpath) and constructor kwargs.
2. queue_to_coordinator — a JSON object mapping a task queue to a coordinator name.

A task whose stub sets queue="..." is handed to the named coordinator, which launches the native subprocess. The coordinator name is arbitrary — it just has to be the same string in both settings. The queue name must match the queue= set on the Python @task.stub.

Option A: airflow.cfg

[sdk]
coordinators = {
  "go": {
    "classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator",
    "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}
  }
}
queue_to_coordinator = {"golang": "go"}

Option B: environment variables

Each value must be valid one-line JSON. This form is convenient for containers, .env files, Docker Compose, and Helm.

export AIRFLOW__SDK__COORDINATORS='{"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}'
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}'

You can register multiple coordinators at once (e.g. one per language) and map different queues to each.

Per-coordinator options

The classpath and kwargs are specific to each coordinator. Add a subsection here as new language SDKs land.

ExecutableCoordinator (Go and other self-contained-executable SDKs)

• classpath: airflow.sdk.coordinators.executable.ExecutableCoordinator
• Worker runtime: none beyond the bundle itself. The bundle is a self-contained native executable (AFBNDL01), so it needs no language runtime, but it must be built for the worker's OS/arch (a mismatch fails with exec format error).

(Future coordinators — for other languages — will list their own classpath, runtime, and kwargs here.)

Verifying the configuration

1. Confirm the artifact is usable where workers run (for the Go SDK: the packed bundle exists and matches the worker's OS/arch) via astro dev bash or docker compose exec ....
2. Confirm the artifact directory referenced in kwargs (e.g. executables_root) actually contains your artifact on the worker filesystem.
3. Trigger the DAG and open the native task's logs — you should see the subprocess start and your task output.

Troubleshooting

Related Skills

• authoring-language-sdk-tasks: The shared Python-stub pattern and conceptual model.
• authoring-go-sdk-tasks: Go task code and matching Python stubs.
• deploying-go-sdk-bundles: Build/pack the Go bundle and place it where the coordinator scans.
• deploying-airflow: General deployment of Airflow on Astro, Docker Compose, or Kubernetes.