Apache Airflow
The standard workflow orchestration tool for data pipelines — schedules, monitors, and manages complex sequences of data processing tasks as code.
Apache Airflow is the most widely adopted workflow orchestration platform for data engineering. It allows teams to define data pipelines as Python code using a DAG (Directed Acyclic Graph) model, where each node is a task and edges define dependencies and execution order. Airflow schedules runs, retries failures, sends alerts, and provides a web UI for monitoring pipeline health. It is deployed at Airbnb (where it was created), Twitter, Lyft, and most data-mature companies. Airflow fluency is a core expectation for data engineering roles and increasingly appears in analytics engineering and MLOps job postings.
Typical time to job-readiness: ~4 weeks.
Learning Apache Airflow
Beginner
Understand what a DAG is and why orchestration matters — what happens when a data pipeline fails halfway through and needs to retry only the failed steps. Install Airflow with Docker Compose and write a simple DAG with a few Python tasks and a schedule interval.
Intermediate
Airflow operators (PythonOperator, BashOperator, cloud-specific operators for BigQuery/Snowflake), task dependencies (>>), XComs for passing data between tasks, and connection/variable management for secrets. Learn how to trigger backfills and understand catchup behavior.
Advanced
Dynamic DAG generation, TaskFlow API (the modern Pythonic DAG authoring style), Airflow on Kubernetes (KubernetesPodOperator, KEDA autoscaling), and testing DAGs with pytest. Data engineering interviews typically include a pipeline design question — be ready to explain how you'd handle late-arriving data, idempotency, and failure recovery.
Key concepts
- DAG (Directed Acyclic Graph) — defines the pipeline as a graph of tasks with dependencies; must have no cycles
- Operators — the building blocks of tasks (PythonOperator, BashOperator, SQLOperator, cloud-specific operators)
- Scheduler — reads the DAG, determines when to run tasks, and sends work to the executor
- XComs — cross-communication mechanism for passing small data between tasks
- Task retries and sensors — retry failed tasks with delays; sensors wait for external conditions before proceeding
- Idempotency — a well-designed pipeline produces the same result if run multiple times for the same period
Common interview topics
- Explain what a DAG is and how Airflow uses it to schedule work
- What is the difference between an operator and a sensor in Airflow
- How do you pass data between tasks in Airflow
- How would you design an idempotent data pipeline in Airflow
- What happens when a task fails — how does Airflow handle retries