Contains tutorial on getting started in using Apache Airflow to automate pipelines.

• Introduction to Apache Airflow

• Getting Started

• Prerequisites

• Write an Airflow DAG

  1. Import libraries
  2. Configure default arguments
  3. Instantiate DAG object
  4. Configure tasks by instantiating operators
  5. Configure tasks dependencies

• Execute DAGs

• Monitor Workflow Using Airflow UI

• Interact with Docker Container

• References

A slide deck which provides brief introduction on what is Apache Airflow, why use Airflow, where is it suitable to use Airflow, and when to use Airflow can be found here.

We are going to use Docker containers to spin up Airflow container as we also need to spin up container for other services. Follow the steps below to get Apache Airflow up and running using Docker container:

1. Clone this repo and cd into this repo's root.
2. Configure host user id, additional Python packages and directories:

echo -e "AIRFLOW_UID=$(id -u)\n_PIP_ADDITIONAL_REQUIREMENTS=pymongo pandas scikit-learn apache-airflow-providers-mongo" > .env && mkdir -p dags logs plugins

3. Initialize the database and create first user account:

docker-compose up airflow-init

You should see similar message as below that indicates success execution:

airflow-init_1       | User "airflow" created with role "Admin"
airflow-init_1       | 2.2.0
airflow-tutorial_airflow-init_1 exited with code 0

4. Start all services: docker-compose up

5. Visit localhost:8080. The default username and password are both airflow.

6. Spin down the containers by: docker-compose down

To completely remove the containers, volumes with DB data and downloaded images, run: docker-compose down --volumes --rmi all

Alternatively, you can install it via PyPi by following the instructions here.

• Installed Docker [manual]
• Installed Docker-compose [manual]
• At least 4GB memory for Docker Engine

DAG stands for directed acyclic graph, which is a collection of tasks which constitute a pipeline/workflow. The steps below are required to construct a DAG.

• Step 1: Import Libraries
• Step 2: Configure default arguments
• Step 3: Instantiate DAG object
• Step 4: Configure tasks by instantiating operators
• Step 5: Configure tasks dependencies

To execute or trigger a DAG manually, first unpause a DAG.

Next, click on the DAG and the "play" button on top right.

The UI will display the execution status after that.

To check out on the details of tasks, click on the task shape in the UI.

Then, select the details which you would like to check out. Log and graph are among useful details which you should check out.

You might want to interact with the Airflow environment. To do so, execute the following and copy the container ID for airflow-worker service:

docker ps

Then run the following to get a bash terminal (remember to replace the right container ID):

docker exec -it <CONTAINER_ID> bash

You can then inspect the environment, for example we check all the DAGs available by:

ls -ltr dags/

• Airflow tutorial 4: Writing your first pipeline
• Running Airflow in Docker
• How to Run Your First Airflow DAG in Docker
• Tutorial
• PostgreSQL Connection
• Psycopg2: PostgreSQL & Python the Old Fashioned Way
• Managing Connections
• Keep TFIDF result for predicting new content using Scikit for Python
• Multi-Class Text Classification with Scikit-Learn
• MongoDB 3.2 authentication failed
• Airflow Schedule Interval 101