Comments (8)
entechlog
commented on July 21, 2024
1
@dweaver33 I will look into this and get back.
from dbt-snow-mask.
dweaver33
commented on July 21, 2024
Any ideas? I still can't get it to apply via dbt. I can do it manually just fine and everything works.
from dbt-snow-mask.
dweaver33
commented on July 21, 2024
Here's a log I took of a recent run:
============================== 2022-07-11 16:07:26.960206 | d30fe986-f225-4f85-b47e-df1d2cde777a ==============================
16:07:26.960251 [info ] [MainThread]: Running with dbt=1.1.1
16:07:26.960917 [debug] [MainThread]: running dbt with arguments {'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/Users/******/.dbt', 'send_anonymous_usage_stats': True, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'macro': 'apply_masking_policy', 'args': '{}', 'which': 'run-operation', 'rpc_method': 'run-operation', 'indirect_selection': 'eager'}
16:07:26.961086 [debug] [MainThread]: Tracking: tracking
16:07:26.961457 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136c8a760>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136c8af70>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136c8a640>]}
16:07:27.017520 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
16:07:27.017715 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
16:07:27.022305 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'd30fe986-f225-4f85-b47e-df1d2cde777a', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136ff00d0>]}
16:07:27.028450 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': 'd30fe986-f225-4f85-b47e-df1d2cde777a', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136dde7f0>]}
16:07:27.028834 [debug] [MainThread]: Acquiring new snowflake connection "macro_apply_masking_policy"
16:07:27.052003 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x137107cd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x137107d90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x137107df0>]}
16:07:27.258886 [debug] [MainThread]: Connection 'macro_apply_masking_policy' was properly closed.
from dbt-snow-mask.
dweaver33
commented on July 21, 2024
@entechlog, have you had a chance to look into this? I still can't get it to apply the policies.
from dbt-snow-mask.
entechlog
commented on July 21, 2024
@dweaver33 , Apologies for the delay. I could not check this out yet. Have been busy with work and personal things. I will try to find sometime this weekend OR next. In the meantime if you have identified the issue please feel free to submit a PR with the fix
from dbt-snow-mask.
dweaver33
commented on July 21, 2024
Thank you! Unfortunately I've been unable to ID what the issue might be.
from dbt-snow-mask.
entechlog
commented on July 21, 2024
I looked into your issue and it seems you are missing a post-hook in your demo dbt_project.yml. It should look something like this. This is because apply for models happens after each individual model finishes currently
models:
post-hook:
- "{{ dbt_snow_mask.apply_masking_policy('models') }}"
from dbt-snow-mask.
dweaver33
commented on July 21, 2024
Ah! Ok, for future folks, here's what I'm doing in my dbt_project.yml:
models:
data_pii_testing:
pii_models:
+persist_docs:
relation: true
columns: true
post-hook:
- "{{ dbt_snow_mask.apply_masking_policy('models') }}"
on-run-start:
- "{{ dbt_snow_mask.unapply_masking_policy('sources')}}"
- "{{ dbt_snow_mask.unapply_masking_policy('models')}}"
- "{{ dbt_snow_mask.create_masking_policy('sources')}}"
- "{{ dbt_snow_mask.create_masking_policy('models')}}"On start, it will unapply the masking policies, then re-create them. This allows you to make updates to the masking policy macros if needed.
Then at the end of each model directory, we apply the masking policies with the post-hook.
from dbt-snow-mask.
Related Issues (20)
- Improvement: include sourcename or modelname in posthook to minimise overhead on run model tasks HOT 2
- apply policy to all models HOT 6
- Apply masking policy with FORCE keyword HOT 5
- Ability to update existing masking functions HOT 2
- Improvement: Allow for default dbt schema name generation HOT 2
- Improvement/bug: alert or warn when a masking policy specified does not exist
- dbt snow mask versioning HOT 2
- Feature Request: Make log optional
- Hyperlink to dbt's meta documetation HOT 1
- Post-hook executed as a pre-hook HOT 2
- Feature Request: Conditional Masking Policy HOT 3
- Applying policy on tables
- Masking policy does not apply for models or snapshots HOT 4
- Conditional Masking Policy does not need `{{masked_column}}` in its body
- Insufficient privilege to create a masking policy
- Unapply policies should handle if the materialization exists or not
- on-run-start with 'models' parameter failed HOT 1
- Abillity to use macro on dynamic talbe
- Ability to pass in node and custom_schema_name HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-snow-mask.