Comments (4)
@belaidcherfa Wow, this is awesome! Thanks for all the work you've put into researching this ❤️ I have totally overlooked the performance part and I really appreciate your help.
Implementing this is straightforward. Do you want to have a crack at it and create a PR? I can also do it, no problem.
from dbt-profiler.
I enjoyed doing the tests.
I'm not used to do PR so if you can do it, that would be great.
Here is the modification I made directly to the package and it works on snowflake. I ran it on several models (different types like tables and views as well as sizes in terms of columns and data).
get_profile_opt.txt
I'm going to take some time over the next few weeks to work on it and try to make the most of the optimizations offered by snowflake in particular.
In terms of caches and metadata : Get the count data separately and Retrieve the min and max separately (in the CTEs apart, it takes them directly from the service layer metadata and it does not query the storage layer ==> So less compute)
from dbt-profiler.
Your proposed improvement has now been implemented and included in the 0.4.1 release 🎉
from dbt-profiler.
I'm excited to see what other Snowflake optimizations you come up with! 💯
from dbt-profiler.
Related Issues (20)
- Include/exclude metrics
- Is this stable? HOT 1
- Question: running dbt run-operation print_profile_docs prints log info HOT 2
- Question: Are you planning to extend dbt-profiler for dbt-sqlserver adapter? HOT 2
- Using print_profile on dbt sources HOT 4
- Allow user to specify a where clause for the source table
- Question: add TOP x column values and distribution. HOT 3
- Feature request: Handling structs HOT 1
- I have modified some of the macros to work with MySql HOT 1
- Catch-22 with docblock in schema file HOT 4
- Suggestion to modify `sed` to `awk` in update-relation-profile.sh HOT 1
- docblock is not outputted when executing `print_profile_docs` with BigQuery
- Run profiler for all models HOT 2
- Add `print_table` macro that works in dbt Cloud
- Redshift `avg` returning `1` or `0` HOT 3
- Error when profiling a table in Redshift HOT 9
- "Value overflow in a SUM aggregate" with HASH fields HOT 4
- `dbt_profiler.get_profile` doesn't seem to work with views in Redshift HOT 5
- Old version of example output, without median column HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-profiler.