tuva-health / tuva_demo Goto Github PK

A starter dbt project and synthetic claims dataset for trying out the Tuva Project.

License: Apache License 2.0

analytics-engineering bigquery data-analytics data-governance data-lineage data-pipelines data-warehouse dbt healthcare healthcare-analysis

tuva_demo's Introduction

The Tuva Project Demo

🧰 What does this project do?

This demo provides a quick and easy way to run the Tuva Project Package in a dbt project with synthetic data for 1k patients loaded as dbt seeds.

To set up the Tuva Project with your own claims data or to better understand what the Tuva Project does, please review the ReadMe in The Tuva Project package for a detailed walkthrough and setup.

For information on data models and to view the entire DAG check out our dbt Docs.

🔌 Database Support

BigQuery
Redshift
Snowflake

✅ How to get started

Pre-requisites

You have dbt installed and configured (i.e. connected to your data warehouse). If you have not installed dbt, here are instructions for doing so.
You have created a database for the output of this project to be written in your data warehouse.

Getting Started

Complete the following steps to configure the project to run in your environment.

Clone this repo to your local machine or environment.
Update the dbt_project.yml file:
1. Add the dbt profile connected to your data warehouse.
Run dbt deps to install the Tuva Project package.
Run dbt build to run the entire project with the built-in sample data.

🙋🏻‍♀️ How is this project maintained and can I contribute?

Project Maintenance

The Tuva Project team maintaining this project only maintains the latest version of the project. We highly recommend you stay consistent with the latest version.

Contributions

Have an opinion on the mappings? Notice any bugs when installing and running the project? If so, we highly encourage and welcome feedback! While we work on a formal process in Github, we can be easily reached on our Slack community.

🤝 Community

Join our growing community of healthcare data practitioners on Slack!

tuva_demo's People

Contributors

Stargazers

Watchers

Forkers

ryandataengineergit gonzalo933 etaylorslc eeethb cnolanminich villagers-health skachur gordonwillingham bryanpmayfield sanjoybosecs subha-cs 0m-3 mimilabs

tuva_demo's Issues

Errors when running dbt deps

I tried to follow the instructions here to set up the demo but after running dbt deps I run into some errors:

CaughtMacroErrorWithNodeError
Compilation Error in seed terminology__icd_10_cm (seeds/terminology/terminology__icd_10_cm.csv)
  macro 'dbt_macro__snowflake__load_seed' takes not more than 1 argument(s)
  
  > in macro load_seed (macros/load_seed.sql)
  > called by seed terminology__icd_10_cm (seeds/terminology/terminology__icd_10_cm.csv)
CaughtMacroErrorWithNodeError

it seems that there has been some changes in the tuva project macros and the post hooks no longer work?

Changing the version in dbt_packages.yml to:

packages:
  - package: tuva-health/the_tuva_project
    version: [">=0.3.0","<=0.3.16"]

solves the problem. I'm using dbt cloud v1.6

Admission and Discharge Dates

Both admission_date and discharge_date are always null. We need admission and discharge dates for inpatient analytics. An easy option might be to use claim_start_date and claim_end_date for any claim_id that has a DRG. We want to make sure that length of stay, which equals discharge - admit, produces a reasonable average. This number should be between 3 and 6 days probably.

Load Sample Dataset via S3

Desired Behavior: Load 1,000 patient sample dataset directly from S3. Remove ability to load smaller seed files (it's better to just have one way to load things). Remove ability to configure by loading from data shares. One way to load is more simple.

Explore BCDA sandbox data

Explore using the BCDA (beneficiary claims data API) sandbox from CMS. This may be a good replacement for the current demo data.

Rename Tuva Claims Demo project

Rename the Tuva Claims Demo name and project to the_tuva_project_demo and remove unnecessary models.

Rename the Tuva Claims Demo repo
Rename the dbt project
Change the schema for seeds back to raw_data.
Remove the models and rename the seed files to the model names by dropping the "_sample" part.
Update the README
Update marketplace listings once repo is renamed

Update Tuva Claims Demo Tutorial

Here's how the quickstart tutorial should work:

Clone tuva_claims_demo from Github
Configure the dbt project to run in your environment
Import the_tuva_project package
Run dbt build. Describe how this loads a synthetic claims dataset into your data warehouse from S3 and builds the entire Tuva Project
Setup and run tuva_analytics

Add synthetic observation seed

Is your feature request related to a problem? Please describe.
Add observations for vital signs to the synthetic data set. Blood pressure readings are needed to validate the build for "Controlling Blood Pressure" (MIPS CQM 236).

Describe the solution you'd like
Populate the observation table in the input layer with systolic and diastolic blood pressure readings.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Refactor Tuva Claims Demo

Tuva Claims Demo is currently not working. We need it working with the refactor.

Aaron's thoughts:

It should include 3 models named after the input layer so the tuva package refs will work.

dispatch in dbt_project.yml wrong

The dispatch variable/function (not sure what to call it) does not work as intended because the 'search order' part is wrong. Instead of 'the_tuva_project' it needs to be the current project's name so it should be 'tuva_claims_demo'.

I changed this locally and confirmed that the issue was fixed. The schemas being generated were no longer prefixed with the target schema.

upgrade tuva_project version to 0.2.3

packages.yml is currently 0.2.1

Ephemeral Model Error w/Postgres

Describe the bug
I have successfully installed the tuva project demo and used dbt build to load the demo data into snowflake. When I try to do this with a local postgres DB, however, I get the following errors after running dbt build:

19:20:32  Finished running 76 view models, 86 seeds, 240 table models, 57 data tests in 0 hours 0 minutes and 11.71 seconds (11.71s).
19:20:32  
19:20:32  Completed with 53 errors and 0 warnings:
19:20:32  
19:20:32    Database Error in model core__stg_clinical_condition (models/core/staging/core__stg_clinical_condition.sql)
  operator does not exist: date ~ text
  LINE 15:       when recorded_date similar to '[0-9]{4}-[0-9]{2}-[0-9]...
                                    ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_condition.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_encounter (models/core/staging/core__stg_clinical_encounter.sql)
  operator does not exist: date ~ text
  LINE 14:       when encounter_start_date similar to '[0-9]{4}-[0-9]{2...
                                           ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_encounter.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_medication (models/core/staging/core__stg_clinical_medication.sql)
  operator does not exist: date ~ text
  LINE 14:       when dispensing_date similar to '[0-9]{4}-[0-9]{2}-[0-...
                                      ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_medication.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_patient (models/core/staging/core__stg_clinical_patient.sql)
  operator does not exist: date ~ text
  LINE 16:       when birth_date similar to '[0-9]{4}-[0-9]{2}-[0-9]{2}...
                                 ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_patient.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_procedure (models/core/staging/core__stg_clinical_procedure.sql)
  operator does not exist: date ~ text
  LINE 15:       when procedure_date similar to '[0-9]{4}-[0-9]{2}-[0-9...
                                     ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_procedure.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_lab_result (models/core/staging/core__stg_clinical_lab_result.sql)
  operator does not exist: date ~ text
  LINE 25:       when result_date similar to '[0-9]{4}-[0-9]{2}-[0-9]{2...
                                  ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_lab_result.sql
19:20:32  
19:20:32    Database Error in model core__stg_clinical_observation (models/core/staging/core__stg_clinical_observation.sql)
  operator does not exist: date ~ text
  LINE 16:       when observation_date similar to '[0-9]{4}-[0-9]{2}-[0...
                                       ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_clinical_observation.sql
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__medication_patient_id.0aaf3a0652, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__medication
19:20:32  
19:20:32    Database Error in model data_quality__claims_preprocessing_eligibility_denominators (models/data_quality/claims_preprocessing/intermediate/data_quality__claims_preprocessing_eligibility_denominators.sql)
  syntax error at or near ")"
  LINE 33: )
           ^
  compiled Code at target/run/the_tuva_project/models/data_quality/claims_preprocessing/intermediate/data_quality__claims_preprocessing_eligibility_denominators.sql
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.hcc_suspecting__int_prep_egfr_labs, caused by compilation error in referenced ephemeral model model.the_tuva_project.hcc_suspecting__stg_core__lab_result
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__observation_patient_id.31e33f2d7e, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__observation
19:20:32  
19:20:32    Database Error in model data_quality__claims_preprocessing_medical_claim_denominators (models/data_quality/claims_preprocessing/intermediate/data_quality__claims_preprocessing_medical_claim_denominators.sql)
  syntax error at or near ")"
  LINE 56: )
           ^
  compiled Code at target/run/the_tuva_project/models/data_quality/claims_preprocessing/intermediate/data_quality__claims_preprocessing_medical_claim_denominators.sql
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_missing_age, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_missing_gender, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.cms_hcc__int_members, caused by compilation error in referenced ephemeral model model.the_tuva_project.cms_hcc__stg_core__patient
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.accepted_values_quality_measures__stg_core__patient_sex__female__male__unknown.e6adc9ce2b, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__patient
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__patient_birth_date.4b3a2e74e6, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__patient
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__patient_patient_id.5510e1a0fd, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__patient
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__patient_sex.e2f34cab69, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.hcc_suspecting__int_all_medications, caused by compilation error in referenced ephemeral model model.the_tuva_project.hcc_suspecting__stg_core__medication
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_01_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_03_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_05_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_07_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_08_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_11_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_12_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_14_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_15_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_16_denom, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_patient
19:20:32  
19:20:32    Database Error in model core__stg_claims_condition (models/core/staging/core__stg_claims_condition.sql)
  operator does not exist: date ~ text
  LINE 503:       when unpivot_cte.recorded_date similar to '[0-9]{4}-[0...
                                                 ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_claims_condition.sql
19:20:32  
19:20:32    Database Error in model core__stg_claims_medical_claim (models/core/staging/core__stg_claims_medical_claim.sql)
  operator does not exist: date ~ text
  LINE 36:       when med.claim_start_date similar to '[0-9]{4}-[0-9]{2...
                                           ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_claims_medical_claim.sql
19:20:32  
19:20:32    Database Error in model core__stg_claims_procedure (models/core/staging/core__stg_claims_procedure.sql)
  operator does not exist: date ~ text
  LINE 511:       when unpivot_cte.procedure_date similar to '[0-9]{4}-[...
                                                  ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_claims_procedure.sql
19:20:32  
19:20:32    Database Error in model core__stg_claims_encounter (models/core/staging/core__stg_claims_encounter.sql)
  operator does not exist: date ~ text
  LINE 25:       when encounter_start_date similar to '[0-9]{4}-[0-9]{2...
                                           ^
  HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
  compiled Code at target/run/the_tuva_project/models/core/staging/core__stg_claims_encounter.sql
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ccsr__long_condition_category, caused by compilation error in referenced ephemeral model model.the_tuva_project.ccsr__stg_core__condition
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.chronic_conditions__cms_chronic_conditions_hiv_aids, caused by compilation error in referenced ephemeral model model.the_tuva_project.cms_chronic_conditions__stg_core__condition
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.hcc_suspecting__int_prep_conditions, caused by compilation error in referenced ephemeral model model.the_tuva_project.hcc_suspecting__stg_core__condition
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__condition_patient_id.1f1222e348, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__condition
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.chronic_conditions__tuva_chronic_conditions_long, caused by compilation error in referenced ephemeral model model.the_tuva_project.tuva_chronic_conditions__stg_core__condition
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.cms_hcc__int_eligible_conditions, caused by compilation error in referenced ephemeral model model.the_tuva_project.cms_hcc__stg_core__medical_claim
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.financial_pmpm__patient_spend_with_service_categories, caused by compilation error in referenced ephemeral model model.the_tuva_project.financial_pmpm__stg_medical_claim
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.quality_measures__int_shared_exclusions_institutional_snp, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_medical_claim
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ccsr__long_procedure_category, caused by compilation error in referenced ephemeral model model.the_tuva_project.ccsr__stg_core__procedure
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.chronic_conditions__cms_chronic_conditions_all, caused by compilation error in referenced ephemeral model model.the_tuva_project.cms_chronic_conditions__stg_core__procedure
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__procedure_patient_id.0b52ff75d9, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__procedure
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.readmissions__procedure_ccs, caused by compilation error in referenced ephemeral model model.the_tuva_project.readmissions__stg_core__procedure
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_missing_dates, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_inpatient_encounter
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_missing_primary_dx, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_inpatient_encounter
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_transfer, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_inpatient_encounter
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ahrq_measures__int_pqi_shared_exclusion_ungroupable_drg, caused by compilation error in referenced ephemeral model model.the_tuva_project.ahrq_measures__stg_pqi_inpatient_encounter
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.ed_classification__int_map_primary_dx, caused by compilation error in referenced ephemeral model model.the_tuva_project.ed_classification__stg_encounter
19:20:32  
19:20:32    Compilation Error in test.the_tuva_project.not_null_quality_measures__stg_core__encounter_patient_id.a0d82e2159, caused by compilation error in referenced ephemeral model model.the_tuva_project.quality_measures__stg_core__encounter
19:20:32  
19:20:32    Compilation Error in model.the_tuva_project.readmissions__encounter, caused by compilation error in referenced ephemeral model model.the_tuva_project.readmissions__stg_core__encounter

To Reproduce
Steps to reproduce the behavior:

In the profiles yaml file, create a profile for connecting to a local postgres install.
run dbt build
See errors as listed above
Note that the database tables are created, but they have no data.

Expected behavior
I expected that database tables in my local postgres instance to have the same demo data as I saw in Snowflake.

Environment

Package version (e.g. 0.6.0):
Data warehouse (e.g. Snowflake): postgres

Additional context
dbt version 1.8.1

Eligibility seed file not loading in Databricks because of datetime data type issue

Describe the bug
Eligibility seed file not loading in Databricks because DATETIME data type is not supported.

To Reproduce
Run Tuva Demo on Databricks.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.