nahomneg / sensor-data-elt Goto Github PK

A fully dockerized ELT pipeline project, using Postgres, MYSQL, dbt, Apache Airflow, Redash and Superset.
Explore the docs »

View Demo · Report Bug · Request Feature

A fully dockerized using a docker-compose file ELT pipeline using MySQL, PostgreSQL, Airflow, DBT, Redash and Superset. used MySQL and Postgres for data ware house. used DBT for data transforming and airflow for automation and orchestrations. A redash and Superset dasboard is built by connecting it to our data ware .

Tech Stack used in this project

• Postgres
• MYSQL
• Apache Airflow
• dbt
• Redash
• Superset

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

1. Clone the repo

   git clone https://github.com/daniEL2371/sensor-data-ELT.git

2. Run

    docker-compose build
    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger load_data dag
   activate and trigger migrate_data dag
   activate and trigger dbt_dbt_dag

4. Access your Postgres database using adminar

   Navigate to `http://localhost:8080/` on the browser
   use `postgres` databse
   use `postgres-dbt` server
   use `dbtuser` for username
   use `pssd` for password

5. Access redash dashboard

   open new terminal
   docker-compose run — rm server create_db
   Open pstgres-dbt using adminar
   Create a user for analytics database
   CREATE USER 'redash'@'' IDENTIFIED WITH mysql_native_password BY 'root';
   Grant all privilages for analytics database
   GRANT ALL PRIVILEGES ON analytics.* To 'redash'@'';

   Navigate to `http://localhost:5000/` on the browser
   Login and use your created user on analytics databse to connect to your databse

6. Access Superset dashboard

   Navigate to `http://localhost:8088/` on the browser
   use `root` for username
   use `roor` for password

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Distributed under the MIT License. See LICENSE for more information.

Daniel Zelalem - [email protected]

• 10 Academy