Apache Airflow

Category	Badges
License
PyPI
Containers
Community
Dev tools

Version	Build Status
Main
3.x

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows (Dags) that orchestrate tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on Dags a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Table of contents

• Project Focus
• Principles
• Requirements
• Getting started
• Installing from PyPI
• Installation
• Official source code
• Convenience packages
• User Interface
• Semantic versioning
• Version Life Cycle
• Support for Python and Kubernetes versions
• Base OS support for reference Airflow images
• Approach to dependencies of Airflow
• Contributing
• Community standards
• Agent-assisted contribution (apache-magpie)
• Voting Policy
• Who uses Apache Airflow?
• Who maintains Apache Airflow?
• What goes into the next release?
• Can I use the Apache Airflow logo in my presentation?
• Links
• Sponsors

Project Focus

Airflow works best with workflows that are mostly static and slowly changing. When the Dag structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include Luigi, Oozie and Azkaban.

Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's XCom feature). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work.

Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.

Principles

• Dynamic: Pipelines are defined in code, enabling dynamic dag generation and parameterization.
• Extensible: The Airflow framework includes a wide range of built-in operators and can be extended to fit your needs.
• Flexible: Airflow leverages the Jinja templating engine, allowing rich customizations.

Requirements

Apache Airflow is tested with:

	Main version (dev)	Stable version (3.3.0)	Deprecate version (2.11.2)
Python	3.10, 3.11, 3.12, 3.13, 3.14	3.10, 3.11, 3.12, 3.13, 3.14	3.10, 3.11, 3.12
Platform	AMD64/ARM64	AMD64/ARM64	AMD64/ARM64(*)
Kubernetes	1.30, 1.31, 1.32, 1.33, 1.34, 1.35	1.30, 1.31, 1.32, 1.33, 1.34, 1.35	1.26, 1.27, 1.28, 1.29, 1.30
PostgreSQL	14, 15, 16, 17, 18	14, 15, 16, 17, 18	12, 13, 14, 15, 16
MySQL	8.0, 8.4, Innovation	8.0, 8.4, Innovation	8.0, Innovation
SQLite	3.15.0+	3.15.0+	3.15.0+

* Experimental

Note: MariaDB is not tested/recommended.

Note: SQLite is used in Airflow tests. Do not use it in production. We recommend using the latest stable version of SQLite for local development.

Note: Airflow currently can be run on POSIX-compliant Operating Systems. For development, it is regularly tested on fairly modern Linux Distros and recent versions of macOS. On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. The work to add Windows support is tracked via #10388, but it is not a high priority. You should only use Linux-based distros as "Production" execution environment as this is the only environment that is supported. The only distro that is used in our CI tests and that is used in the Community managed DockerHub image is Debian Bookworm.

Getting started

Visit the official Airflow website documentation (latest stable release) for help with installing Airflow, [getting started](https:/