The paradime-dbt from vasiliisurov

Create a new gcp account and activate 90 days trial
```
[email protected]
boxwood-academy-373200
```

Switch default project to the newly created one

gcloud auth login
gcloud config configurations list
gcloud config set project boxwood-academy-373200

Create 2 buckets

# boxwood-academy-inbound
# boxwood-academy-archive
# boxwood-academy-source

gsutil mb gs://boxwood-academy-inbound
gsutil mb gs://boxwood-academy-archive
gsutil mb gs://boxwood-academy-source

Create BQ dataset for raw data
```
bq mk raw
```

Create new dbt profile Add new profile (create new)

vi ~/.dbt/profiles.yml

 paradime:
     outputs:
         dev:
         dataset: dev
         fixed_retries: 1
         location: US
         method: service-account
         keyfile: /Users/vasilii.surov/vscode/paradime-dbt/docker/dbt-paradime.json
         priority: interactive
         project: boxwood-academy-373200
         threads: 5
         timeout_seconds: 300
         type: bigquery

         prod:
         dataset: prod
         fixed_retries: 1
         location: US
         method: service-account
         keyfile: /Users/vasilii.surov/vscode/paradime-dbt/docker/dbt-paradime.json
         priority: interactive
         project: boxwood-academy-373200
         threads: 5
         timeout_seconds: 300
         type: bigquery

 target: dev

Create and authenticate new dbt project

dbt init --adapter bigquery paradime
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/drive.readonly

Check out dbt configuration
```
dbt debug
```
Initialize git
```
git init
```
dbt-utils package
```
dbt deps
```

Download all csv data locally

/home/sova/.venv/dbt/bin/python3 /home/sova/IdeaProjects/dbt/sunrun/scripts/download.py

Recompressing with gzip
```
unzip \*.zip
gzip *.csv
```

Upload all files to source bucket

gsutil -m cp *.csv.gz gs://boxwood-academy-source/citibike/

Load test volume of data into inbound bucket

gsutil -m cp gs://boxwood-academy-source/citibike//2018*.gz gs://boxwood-academy-inbound/citibike/

Load files from gcs into BQ
```
./scripts/bq_load.sh
```
Build dbt models: chains start [end == start] [end == start] .. end
Composer jobs dbt_compile.py
Models: see dbt docs
```
dbt docs generate
dbt docs serve
```

vasiliisurov / paradime-dbt Goto Github PK

paradime-dbt's Introduction

paradime-dbt's People

Contributors

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent