CTE Query Optimization about dbt-profiler HOT 4 CLOSED

@belaidcherfa Wow, this is awesome! Thanks for all the work you've put into researching this ❤️ I have totally overlooked the performance part and I really appreciate your help.

Implementing this is straightforward. Do you want to have a crack at it and create a PR? I can also do it, no problem.

from dbt-profiler.

I enjoyed doing the tests.
I'm not used to do PR so if you can do it, that would be great.
Here is the modification I made directly to the package and it works on snowflake. I ran it on several models (different types like tables and views as well as sizes in terms of columns and data).
get_profile_opt.txt

I Added this here :

And change this :

I'm going to take some time over the next few weeks to work on it and try to make the most of the optimizations offered by snowflake in particular.
In terms of caches and metadata : Get the count data separately and Retrieve the min and max separately (in the CTEs apart, it takes them directly from the service layer metadata and it does not query the storage layer ==> So less compute)

from dbt-profiler.

Your proposed improvement has now been implemented and included in the 0.4.1 release 🎉

from dbt-profiler.

I'm excited to see what other Snowflake optimizations you come up with! 💯

from dbt-profiler.