This repository shows you the potential stack of technologies and their cooperation to conduct experiments on data pipelines.
The experiment is a change in methodology and obtaining final results based on the impact of these changes.
- Install SBT.
- Open the project in IntelliJ IDEA or something similar.
- Setup SDK for Java and Scala.
- Import jobs directory as SBT module.
- Generate jar file
cd ./jobs/ && sbt assembly
- Go to ./service/zeppelin and execute:
docker build . -t zeppelin-spark-3
- Go to ./service/airflow and execute:
docker build . -t airflow
- Go to ./service and execute:
docker-compose up