Apache Airflow
An open source platform for building, scheduling and monitoring batch data workflows written in Python.
Apache Airflow is a workflow platform. You describe a series of tasks in Python, set the order they run in and Airflow handles scheduling them, running them and showing you what happened. The official documentation calls it a platform for developing, scheduling and monitoring batch oriented workflows.
What is Apache Airflow?
Apache Airflow is an open source platform for developing, scheduling and monitoring batch workflows. It is written in Python, and the workflows you give it are written in Python as well.
A workflow in Airflow is called a DAG, short for directed acyclic graph. The name only means a set of tasks with a defined order and no loops. Airflow also ships with a web interface that shows every workflow, every run of that workflow and the status of each task inside it.
What does Apache Airflow do?
Airflow does not process data itself. It decides what runs, when it runs and what has to finish first. The official documentation gives typical work as running a Spark job, moving files between storage buckets or sending a notification email.
Airflow also keeps a record of past runs. If a task failed last night you can open that run in the interface, read the log and start the task again. When you need to process a range of past dates, Airflow calls that a backfill, and it uses the same code as the scheduled run.
How does Apache Airflow work?
- You write a Python file that defines a DAG. The file lists the tasks, the order between them and the schedule.
- Airflow reads the file and registers the DAG.
- The scheduler creates a run for the DAG at each scheduled time.
- Workers pick up individual tasks and execute them in the order you defined.
- The web interface shows each task as it moves from queued to running to success or failure.
Because a DAG is ordinary Python, the tools you already use for code apply to it. Files sit in version control and can be reviewed and tested.
Who can use Apache Airflow?
Airflow is built for people who are comfortable writing Python. The documentation states this plainly. The platform suits users who prefer coding over clicking, and teams that want version control, testing and collaboration around their workflows.
It also scales in both directions. The documentation notes that Airflow can run as a single process on a laptop or as a distributed system handling large workloads.
Limitations to keep in mind
Airflow is built for batch workflows, which the documentation defines as workflows with a clear start and end that run on a schedule. It is not designed for continuous stream processing.
Writing code is not optional either. The documentation says Airflow is not a good fit if you prefer clicking over coding, because defining workflows in Python is a core design principle rather than one option among several.
Getting started with Apache Airflow
The official documentation provides a Quick Start page, an installation guide and a set of tutorials. A first DAG can be written with the DAG decorator and a task operator, both shown in the documentation examples.
Once Airflow is running locally, open the web interface, trigger the DAG by hand and watch each task change state.
Key features of Apache Airflow
Capabilities described in the official documentation.
Workflows written in Python
Pipelines are defined in Python code, which allows workflows to be generated dynamically and parameterised.
A large set of built in operators
The platform ships with a wide range of operators and can be extended when a task is not already covered.
Jinja templating
Airflow uses the Jinja templating engine, so values such as dates can be placed into tasks at run time.
Web interface for running and inspecting DAGs
The interface lets you trigger a DAG by hand, inspect a run and process historical periods through a backfill.
Advantages of Apache Airflow
Factual advantages that follow from the features above.
Workflows live in version control
Because a DAG is a Python file, it can be reviewed, tested and tracked with the same tools as any other code.
Past runs stay visible
Every run is recorded with its task states and logs, so a failed step can be found and started again from the interface.
One platform from laptop to cluster
The documentation states that Airflow can run as a single process for development or as a distributed system for larger workloads.
Dependencies are stated, not assumed
Because each task declares what has to finish first, a step cannot start before the work it depends on has completed.
Common use cases for Apache Airflow
Situations the official documentation describes this tool as being used for.
Running Spark jobs
Airflow starts a Spark job as a task and waits for it to finish before the next task begins.
Moving files between storage buckets
A task copies or moves files between object storage locations as one step of a scheduled workflow.
Reprocessing historical dates
A backfill runs the same workflow over a past date range, which the documentation lists as a standard operation.
Coordinating steps across systems
One workflow calls jobs in several different systems and holds them in the order the file defines.
Official website
Everything on this page is based on the official documentation for Apache Airflow. You can read the source here.
Apache Airflow official documentation