Most of tutorials just explains how to use the Airflow DockerOperator using the bare metal installation; and here we will use it with Airflow on top of Docker Compose.

• @fclesio

1. First create a container with the webservice and create the airflow user, as described in the official docs:

$ docker-compose up airflow-init

or execute the following script below:

$ bash bin/initial_setup.sh

2. With this initial setup made, start the webservice and other components via docker-compose,

$ docker build -f dags/docker_job/Dockerfile -t docker_image_task . && \
docker-compose up -d

or execute the following script below that will do the same thing:

$ bash bin/start.sh

3. Finally when you're done with your experiment, stop all containers running the following command:

$  bash bin/stop.sh

Useful links that I used during this endeavor in importance order.

• Running Airflow in Docker

• How can I used the DockerOperator in Airflow, of I am already running Airflow in Docker?

• Using Docker-in-Docker for your CI or testing environment? Think twice.

• How to use the DockerOperator in Apache Airflow

• How to Use Airflow without Headaches

• Docker Thanos (use it carefully)

• DockerOperator Could not serialize the XCom value into JSON

• Is it ok to run docker from inside docker?

• chmod not working correctly in Docker

• Airflow DockerOperator: connect sock.connect(self.unix_socket) FileNotFoundError: [Errno 2] No such file or directory

• How to mount docker socket as volume in docker container with correct group

• Permission issue on running docker command in Python subprocess through Apache Airflow

• How To Run Docker in Docker Container