Code Monkey home page Code Monkey logo

dbt-osmosis's Introduction

dbt-osmosis

PyPI Downloads License: Apache 2.0 black Streamlit App

Scc Count Badge Scc Count Badge

New to dbt-osmosis?

We now have a spiffy dbt-osmosis documentation site! ๐ŸŽ‰

Please check it out for a more in-depth introduction to dbt-osmosis. ๐Ÿ‘‡

dbt-osmosis

What is dbt-osmosis?

Hello and welcome to the project! dbt-osmosis ๐ŸŒŠ serves to enhance the developer experience significantly. We do this through providing 4 core features:

  1. Automated schema YAML management.

    1a. dbt-osmosis yaml refactor --project-dir ... --profiles-dir ...

    Automatically generate documentation based on upstream documented columns, organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules, inject columns from data warehouse schema if missing in yaml and remove columns no longer present in data warehouse (organize -> document)

    1b. dbt-osmosis yaml organize --project-dir ... --profiles-dir ...

    Organize yaml files based on configurable rules defined in dbt_project.yml, scaffold new yaml files based on the same rules

    1c. dbt-osmosis yaml document --project-dir ... --profiles-dir ...

    Automatically generate documentation based on upstream documented columns

  2. A highly performant dbt server which integrates with tools such as dbt-power-user for VS Code to enable interactive querying + realtime compilation from your IDE

    2a. dbt-osmosis server serve --project-dir ... --profiles-dir ...

    Spins up a WSGI server. Can be passed --register-project to automatically register your local project

  3. Workbench for dbt Jinja SQL. This workbench is powered by streamlit and the badge at the top of the readme will take you to a demo on streamlit cloud with jaffle_shop loaded (requires extra pip install "dbt-osmosis[workbench]").

    3a. dbt-osmosis workbench --project-dir ... --profiles-dir ...

    Spins up a streamlit app. This workbench offers similar functionality to the osmosis server + power-user combo without a reliance on VS code. Realtime compilation, query execution, pandas profiling all via copying and pasting whatever you are working on into the workbenchat your leisure. Spin it up and down as needed.

  4. Diffs for data model outputs to model outputs across git revisions (๐Ÿšง this is in development)

    4a. dbt-osmosis diff -m some_model --project-dir ... --profiles-dir ...

    Run diffs on models dynamically. This pulls the state of the model before changes from your git history, injects it as a node to the dbt manifest, compiles the old and modified nodes, and diffs their query results optionally writing nodes to temp tables before running the diff query for warehouses with performance or query complexity limits (๐Ÿ‘€ bigquery)


Pre-commit

You can use dbt-osmosis as a pre-commit hook. This will run the dbt-osmosis yaml refactor command on your models directory before each commit. This is one way to ensure that your schema.yml files are always up to date. I would recommend reading the docs for more information on what this command does.

repos:
  - repo: https://github.com/z3z1ma/dbt-osmosis
    rev: v0.11.11 # verify the latest version
    hooks:
      - id: dbt-osmosis
        files: ^models/
        # you'd normally run this against your prod target, you can use any target though
        args: [--target=prod]
        additional_dependencies: [dbt-<adapter>]

Workbench

The workbench is a streamlit app that allows you to work on dbt models in a side-by-side editor and query tester. I've kept this portion of the README since users can jump into the streamlit hosted workbench to play around with it via the badge below. Expect the living documentation moving forward to exist at the dbt-osmosis documentation site.

I also expect there is some untapped value in the workbench that is only pending some time from myself. I've seen a path to a truly novel development experience and look forward to exploring it.

Demo the workbench ๐Ÿ‘‡

Streamlit App

# NOTE this requires the workbench extra as you can see
pip install "dbt-osmosis[workbench]"

# Command to start server
dbt-osmosis workbench

Press "r" to reload the workbench at any time.

โœ”๏ธ dbt Editor with instant dbt compilation side-by-side or pivoted

โœ”๏ธ Full control over model and workbench theme, light and dark mode

โœ”๏ธ Query Tester, test the model you are working on for instant feedback

โœ”๏ธ Data Profiler (leverages pandas-profiling)

Editor

The editor is able to compile models with control+enter or dynamically as you type. Its speedy! You can choose any target defined in your profiles yml for compilation and execution.

editor

You can pivot the editor for a fuller view while workbenching some dbt SQL.

pivot

Test Query

Test dbt models as you work against whatever profile you have selected and inspect the results. This allows very fast iterative feedback loops not possible with VS Code alone.

test-model

Profile Model Results

Profile your datasets on the fly while you develop without switching context. Allows for more refined interactive data modelling when dataset fits in memory.

profile-data

Useful Links and RSS Feed

Some useful links and RSS feeds at the bottom. ๐Ÿค“

profile-data


graph

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.