Apache Airflow

Three strengths

• Workflows as code. They're defined in Python, so they can be version controlled, tested and worked on as a team. 1
• Scheduling you can see. Workflows run on a schedule, historical data can be backfilled, and only failed steps need rerunning, with a web screen showing logs and status. 1
• Connects to many tools. A large set of built-in operators and an extensible design let it work with many other technologies. 1

What it is

Behind most reports sits a chain of jobs: collect the data, clean it, combine it, load it, refresh the numbers. These have to happen in the right order, at the right time, and someone has to know when one fails. That job is called orchestration, and Apache Airflow is a tool for it. It's an open source platform for developing, scheduling and monitoring workflows. 1

In Airflow, a workflow is called a DAG, short for directed acyclic graph. In Airflow's words, it's "a model that encapsulates everything needed to execute a workflow": when it runs, which tasks it contains, and the order and conditions for running them. 1 Because workflows are written in Python and not clicked together in a screen, they can be kept under version control and reviewed like any other code. 1

Maxime Beauchemin started Airflow at Airbnb in October 2014, and it was open source from the first commit. It was announced in June 2015, joined the Apache Software Foundation's Incubator in March 2016, and became a top-level Apache project in January 2019. 2

Two honest caveats. Airflow is built for batch work, "a platform for orchestrating batch workflows", so it suits scheduled work, not data that has to be handled the instant it arrives. 1 And it takes skill to set up well: a 2024 study of 1,000 Airflow questions from developers found the biggest obstacles arise when defining and running workflows, with incorrect configuration and complex environment setup among the root causes. 3

We choose it because scheduled data work should be visible and dependable. When a job fails at 2am, someone should be able to see what failed and rerun just that step.

How we use it

Airflow carries our scheduled data work. In a Data Solutions project, it runs the scheduled jobs that move and prepare your data, in the right order, and flags any that fail. The heavy processing sits alongside it in Spark. Data that has to arrive the moment it's created goes through WebSockets and serverless event-driven pipelines instead.

That's Data Solutions work, and for you it means reports and dashboards built on data that refreshes on time, every time.

References

1. The Apache Software Foundation, "Apache Airflow Documentation", airflow.apache.org. https://airflow.apache.org/docs/apache-airflow/stable/index.html (accessed 29 October 2026)
2. The Apache Software Foundation, "The Apache Airflow Project", Airflow documentation (apache/airflow on GitHub). https://github.com/apache/airflow/blob/main/airflow-core/docs/project.rst (accessed 29 October 2026)
3. Jerin Yasmin, Jiale Wang, Yuan Tian and Bram Adams, "An Empirical Study of Developers' Challenges in Implementing Workflows as Code: A Case Study on Apache Airflow", arXiv:2406.00180, 2024. https://arxiv.org/abs/2406.00180 (accessed 29 October 2026)