Airflow

From empty directory to a running DAG on local Astro in about fifteen minutes. Assumes less experience with Airflow.

The most common Airflow mistake is treating it as a compute engine. Internalize this one distinction and every other decision follows.

Airflow 3.0 shipped in April 2025 and reshaped the model. The migration reality, the net-new capabilities, and the things that will break your old DAGs.

Three ways to say 'run B after A'. They are not equivalent. The decision framework for which to reach for when.

The idempotency, atomicity, and DAG-shaping rules that let an Airflow pipeline survive contact with a backfill. Walks through a realistic hourly ingest DAG.

The hard part of orchestration is what happens when things go wrong. Retries are for transient failures, pools are for rate-limited resources, sensors are for not-ready-yet. Get the distinction right or you build flakiness.

The on-call procedure for a failed task: classify, check blast radius, pick a recovery path. Five minutes to the first decision.

Stop polling. Trigger DAGs when the data arrives, whether the producer is another DAG or an external system.

The subset of Airflow and Astro CLI commands you use daily, plus the REST API fallback when the CLI does not cover something.

The Causeway rules for a DAG: shape, dependencies, retries, resource gating, deployment. Enforced at review time.

What an Airflow deployment must satisfy before the first production DAG runs, and keep satisfying after. Enforced at the promote-to-prod gate.

The six concurrency knobs, how they interact, and how to tune for the bottleneck actually limiting you.

Symptom-first lookup for the errors on-call hits weekly. Task failures, scheduler issues, resource exhaustion.