Code Monkey home page Code Monkey logo

miconnell / dbt-data-reliability Goto Github PK

View Code? Open in Web Editor NEW

This project forked from elementary-data/dbt-data-reliability

0.0 0.0 0.0 7.64 MB

dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

Home Page: https://www.elementary-data.com/

License: Apache License 2.0

Shell 0.10% Python 96.13% PLpgSQL 3.77%

dbt-data-reliability's Introduction

Logo

License Downloads

What is Elementary?

This dbt package is part of Elementary, the dbt-native data observability solution for data and analytics engineers. Set up in minutes, gain immediate visibility, detect data issues, send actionable alerts, and understand impact and root cause. Available as self-hosted or Cloud service with premium features.

Table of Contents

Quick start - dbt package

  1. Add to your packages.yml:
packages:
  - package: elementary-data/elementary
    version: 0.14.1
    ## Docs: https://docs.elementary-data.com
  1. Run dbt deps

  2. Add to your dbt_project.yml:

models:
  ## elementary models will be created in the schema '<your_schema>_elementary'
  ## for details, see docs: https://docs.elementary-data.com/
  elementary:
    +schema: "elementary"
  1. Run dbt run --select elementary

Check out the full documentation.

Get more out of Elementary dbt package

Elementary has 3 offerings: This dbt package, Elementary Community (OSS) and Elementary (cloud service).

  • dbt package
    • For basic data monitoring and dbt artifacts collection, Elementary offers a dbt package. The package adds logging, artifacts uploading, and Elementary tests (anomaly detection and schema) to your project.
  • Elementary Community
    • An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on Slack if needed.
  • Elementary Cloud
    • Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more, book a demo or start a trial.

Run Results and dbt artifacts

The package automatically uploads dbt artifacts and run results to tables in your data warehouse:

Run results tables:

  • dbt_run_results
  • model_run_results
  • snapshot_run_results
  • dbt_invocations
  • elementary_test_results (all dbt test results)

Metadata tables:

  • dbt_models
  • dbt_tests
  • dbt_sources
  • dbt_exposures
  • dbt_metrics
  • dbt_snapshots

Here you can find additional details about the tables.

Data anomaly detection as dbt tests

Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc. Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.

Elementary tests are configured and executed like native tests in your project!

Example of Elementary test config in properties.yml:

models:
  - name: your_model_name
    config:
      elementary:
        timestamp_column: updated_at
    tests:
      - elementary.table_anomalies
      - elementary.all_columns_anomalies

Read about the available Elementary tests and configuration.

How Elementary works?

Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs.

The cloud service or the CLI tool read the data from these tables, send alerts and present the results in the UI.

Community & Support

Contributions

Thank you 🧡 Whether it’s a bug fix, new feature, or additional documentation - we greatly appreciate contributions!

Check out the contributions guide and open issues in the main repo.

dbt-data-reliability's People

Contributors

elongl avatar maayan-s avatar oravi avatar idoneshaveit avatar haritamar avatar dapollak avatar roitabach avatar ellakz avatar ofek1weiss avatar web-flow avatar noakurman avatar noyaarie avatar github-actions[bot] avatar ekatia avatar smitsrr avatar mossyyy avatar theromsh avatar suelai avatar arun-kc avatar tosborne-slalom avatar ecalzo avatar kouridis avatar civitaspo avatar pei0804 avatar mossyyyy avatar syou6162 avatar frannydelaney avatar tc-chrisbui avatar avivzikeldy avatar kkprab avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.