data-warehouse-with-apache-airflow-redash-and-dbt's Introduction

Data Pipeline and Reporting Automation

Welcome to a project focused on deploying sensors to businesses and collecting data to provide critical intelligence. In this guide, I'll walk you through deploying our data warehouse, orchestration service, ELT tool, and setting up Redash for data visualization.

Core Tasks

Data Warehouse Setup

Clone the repository:

git clone https://github.com/biniyam69/Data-Warehouse-with-Apache-Airflow-Redash-and-DBT-.git datawarehouse
cd datawarehouse

Install required packages:
```
pip install -r requirements.txt
```
Navigate to the Airflow directory and build Docker containers:
```
cd airflow
docker-compose build
```
Start Airflow and the associated services:
```
docker-compose up -d
```
Access Airflow web server at http://localhost:8080.

Redash Setup

Navigate to the Redash directory:
```
cd redash
```

Create an environment file named .env with the following contents:

REDASH_HOST=http://localhost/redash
PYTHONUNBUFFERED=0
REDASH_LOG_LEVEL=INFO
REDASH_REDIS_URL=redis://redis:6379/0
POSTGRES_PASSWORD=password
REDASH_COOKIE_SECRET=redash-selfhosted
REDASH_SECRET_KEY=redash-selfhosted
REDASH_DATABASE_URL={postgresql+psycopg2://username:password@host/dbname}

Replace {postgresql+psycopg2://username:password@host/dbname} with your PostgreSQL database URL.

Create and initialize the Redash database:
```
docker-compose run --rm server create_db
```
Start Redash:
```
docker-compose up -d
```
Access Redash dashboard at http://localhost:5000.