Airflow

Recommendation

Updated

Moved

HOLD

2022-05-13

↓

What is it

Airflow manages and monitors scheduled workflows such as data pipelines. It is configured through a Python SDK and in GCP it's hosted in a Cloud Composer environment.

When to use it

When you want a GUI to monitor scheduling and execution of workflows. However, if a workflow is a batch data pipeline that is just a SQL statement, consider using Dataform instead.

How to learn it

• Read the introduction
• Read about best practices here and here. We try to implement these.

Why it's on hold

Airflow was assessed in early stages of data pipeline development at Einride, but dbt has since become the preferred toolchain for developing and orchestrating data pipelines.