Here in this repository, we have designed a simple ETL process that extract data from an API and we are transforming this data using Spark and loading this data into an AWS S3 bucket. We running this batch processes using Airflow by Spark job submit Operator in Airflow. All the processes described here are happening on a Docker containers. You can look at this repository if you are interested in local deployment as opposed to Docker-based solution.

• Clone the Github repository
• Build the Spark and the Airflow image
• Create your dags, logs, plugins folder
• Create your environment variable
• Start and run the Spark and Airflow containers
• Run your Spark jobs to confirm if the Spark job completed successfully before moving it to Airflow
• Design the Airflow DAG to trigger and schedule the Spark jobs.

git clone https://github.com/yTek01/docker-spark-airflow.git

docker build -f Dockerfile.Spark . -t spark-air

docker build -f Dockerfile.Airflow . -t airflow-spark

mkdir ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env

AIRFLOW_UID=33333
AIRFLOW_GID=0
AWS_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
AWS_SECRET_KEY=XXXXXXXXXXXXXXXXXXXX

docker-compose -f docker-compose.Spark.yaml -f docker-compose.Airflow.yaml up -d

When all the services all started successfully, now go to http://localhost:8080/ to check that Airflow has started successfully, and http://localhost:8090/ that Spark is up and running.

• Run your Spark jobs to confirm if the Spark job completed successfully before moving it to Airflow.

docker exec -it <Spark-Worker-Contianer-name> \
    spark-submit --master spark://XXXXXXXXXXXXXX:7077 \
    spark_etl_script_docker.py

If all is fine with the setup, i.e. the Spark job completed successfully, then move forward to scheduling the Spark job on Airflow.