Before you running this installation, make sure you have Docker and Docker Compose on your Linux/Unix machine.
In this Airflow installation, you will install Airflow 2.2.0 and some services to support Airflow such as :
-
Databases:
- First Postgresql image as datalake
- Second Postgresql image as datawarehouse
-
Airflow Setup:
- Postgresql as Airflow repo
- Redis
- Airflow webserver, scheduler, worker
- Flower
- To run this installation, you must config the credential inside the .env file. For example:
AIRFLOW_UID=50000
DB_NAME=dev
USER_NAME=user_dev
USER_PASSWORD=pass2022!
AIRFLOW_CONN_POSTGRES_DATALAKE=postgresql://user_dev:[email protected]:5433/dev
AIRFLOW_CONN_POSTGRES_DATAWAREHOUSE=postgresql://user_dev:[email protected]:5434/dev
- Change the credential inside the
credentials.json
file in this folderdags/script/
. For example:
{
"postgres_lake": {
"host":"172.17.0.1",
"port":"5433",
"database":"dev",
"username":"user_dev",
"password":"pass2022!"
},
"postgres_warehouse": {
"host":"172.17.0.1",
"port":"5434",
"database":"dev",
"username":"user_dev",
"password":"pass2022!"
}
}
- Then, open your terminal and enter to installation folder where you can find docker-compose.yaml file.
- Run this command to turn on your installation. It will download and install some docker images for 5-20 minutes based on your internet connection.
docker-compose up
- After installation finish, open your browser and type this URL :
localhost:8080
- You need to insert username and password for Airflow for the first time login, please use this credential below.
username : airflow
password : airflow
- You will see a DAG named
postgres2postgres_manual
and you can trigger this DAG using play button on the right page. - Click on
postgres2postgres_manual
DAG and you will see dag tree and wait for the process until finish. - If all process success, you can use your Postgresql client and enter to
postgres_warehouse
database to see table that already created and inserted. - To turn off Airflow installation, you can use this command on your terminal.
docker-compose down