This repository contains code and examples used alongside the Apache Airflow Technical Essentials live training on O'Reilly.
mkdir airflow-training && cd airflow-training
python3 -m venv airflow-venv
source ./airflow-venv/bin/activate
export airflow_python_version=$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)
export constraint_url="https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-${airflow_python_version}.txt"
pip install apache-airflow==2.9.1 --constraint "${constraint_url}"
export AIRFLOW_HOME=$(pwd)
export AIRFLOW__CORE__LOAD_EXAMPLES=False
mkdir dags
cp ../dags/example_dag.py dags/
airflow standalone
echo -e "AIRFLOW_UID=$(id -u)" >> .env
# Initialize the database
docker compose up airflow-init
# Run airflow
docker compose up
# user: airflow
# password: airflow
# cleanup
docker compose down --volumes --remove-orphans
A basic DAG that introduces the concepts of tasks, operators, dependencies, and the DAG.
A more robust DAG that uses variable, connection, xcom, SimpleHTTPOperator, and task context.
A dag that queries the Airflow metastore using the PostgresOperator.
docker exec -it $POSTGRES_CONTAINER psql -h postgres -U airflow
# password is "airflow"
- The logical date, formerly known as execution date, is the start of the data interval, not when the DAG is actually executed.
- Environment variables override
airflow.cfg(See.envfor examples)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent