Comments (5)
@stumelius , the naming convention may be wonky, but information_schema.tables does contain views! My query is just an example of a union that results in VIEW
, BASE TABLE
, and EXTERNAL TABLE
information. If this union, or just information_schema.tables
is okay given the local results in Redshift (pic I shared in this comment), then it will take a bit more testing given that the dtype differences between pg_catalog and information_schema may not function well downstream for dbt-profiler. I'll develop some use cases for each type of database object locally, to test out the union.
from dbt-profiler.
@stevenconnorg Thanks for reporting this and sorry for the late response. I'm okay with the pg_get_cols
approach you described. Would you be willing to contribute the fix? I don't currently have access to a Redshift cluster and I'm therefore unable to test this. I'd appreciate your help :)
from dbt-profiler.
@stevenconnorg and @stumelius , what are your thoughts on adding additional support for external table references in addition to aforementioned views for Redshift? I have a local modification that I've made to dbt_utils package, that would likely also be required to dbt-profiler to take svv_external_schemas into account. Below is a local modification I had to make within a dbt_utils macro. Works well thus far.
with internalDbCTE as (
select distinct
table_schema as {{ adapter.quote('table_schema') }},
table_name as {{ adapter.quote('table_name') }},
{{ dbt_utils__vDGS.get_table_types_sql() }}
from {{ database }}.information_schema.tables
where table_schema ilike '{{ schema_pattern }}'
and table_name ilike '{{ table_pattern }}'
and table_name not ilike '{{ exclude }}'
)
-- following necessary for redshift external tables. May not work for other DBs
, externalDbCTE as (
select distinct
schemaname as {{ adapter.quote('schemaname') }},
tablename as {{ adapter.quote('tablename') }},
'EXTERNAL TABLE' as table_type
from pg_catalog.svv_external_columns
where redshift_database_name ilike '{{ database }}'
and schemaname ilike '{{ schema_pattern }}'
and tablename ilike '{{ table_pattern }}'
and tablename not ilike '{{ exclude }}'
)
select * from internalDbCTE
union all
select * from externalDbCTE
Here's an example of a call on my local DB
from dbt-profiler.
@odikia Thank your for the suggestion. I don't have anything against adding support for external tables. Now, coming back to the issue of views not being included. In the query you provided, there's a union between information_schema.tables
and pg_catalog.svv_external_columns
- neither of these contain views, right? Do you know where we could get the view data? pg_get_cols
works at the relation level and we'd have to loop over all the relations to get a complete picture - not ideal.
from dbt-profiler.
@odikia Got it, thanks for clarifying! Feel free to open a PR when you have something and I'll have a look at it :)
from dbt-profiler.
Related Issues (20)
- Include/exclude metrics
- Is this stable? HOT 1
- CTE Query Optimization HOT 4
- Question: running dbt run-operation print_profile_docs prints log info HOT 2
- Question: Are you planning to extend dbt-profiler for dbt-sqlserver adapter? HOT 2
- Using print_profile on dbt sources HOT 4
- Allow user to specify a where clause for the source table
- Question: add TOP x column values and distribution. HOT 3
- Feature request: Handling structs HOT 1
- I have modified some of the macros to work with MySql HOT 1
- Catch-22 with docblock in schema file HOT 4
- Suggestion to modify `sed` to `awk` in update-relation-profile.sh HOT 1
- docblock is not outputted when executing `print_profile_docs` with BigQuery
- Run profiler for all models HOT 2
- Add `print_table` macro that works in dbt Cloud
- Redshift `avg` returning `1` or `0` HOT 3
- Error when profiling a table in Redshift HOT 9
- "Value overflow in a SUM aggregate" with HASH fields HOT 4
- Old version of example output, without median column HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-profiler.