See

Would like to add run_results.json to the processing engine, including the execution time per model. Have some code written for this locally and would appreciate guidance on how to commit to your repo for review.

Create a new docs folder to consolidate images and diagrams used in documentation

Hey team,
thanks for the great work with olivertwist!

We are using olivertwist in the CircleCI pipeline in one of our dbt projects and it works great. When rolling it out to other dbt projects I encountered an issue. Set up for olivertwist is the same and I am not aware of any difference in the dbt project that could explain this.

I am not sure how to debug this, so wanted to check if you have any ideas or pointers for me. I also had talked to a colleague that had the same issue in a different dbt project, so it seems not just my local set up.

Let me know if there is a better place or way to report issues like this. And also if you would need more information around set up to troubleshoot this.

Thank you!

Version: olivertwist 0.2

Traceback:

Traceback (most recent call last):
  File "/.venv/bin/olivertwist", line 8, in <module>
    sys.exit(main())
  File "/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/.venv/lib/python3.8/site-packages/olivertwist/main.py", line 66, in check
    results = rule_engine.run(manifest)
  File "/.venv/lib/python3.8/site-packages/olivertwist/ruleengine/engine.py", line 44, in run
    return [Result(rule, *rule.apply(manifest)) for rule in self.rules]
  File "/.venv/lib/python3.8/site-packages/olivertwist/ruleengine/engine.py", line 44, in <listcomp>
    return [Result(rule, *rule.apply(manifest)) for rule in self.rules]
  File "/.venv/lib/python3.8/site-packages/olivertwist/ruleengine/rule.py", line 30, in apply
    return self.func(manifest)
  File "/.venv/lib/python3.8/site-packages/olivertwist/rules/no_references_outside_of_its_staging_area.py", line 35, in no_references_outside_of_its_own_staging_area
    return list(passes), list(failures)
  File "/.venv/lib/python3.8/site-packages/olivertwist/rules/no_references_outside_of_its_staging_area.py", line 24, in staging_depends_on_staging_in_another_area
    different_staging_area_refs = [
  File "/.venv/lib/python3.8/site-packages/olivertwist/rules/no_references_outside_of_its_staging_area.py", line 28, in <listcomp>
    if not manifest.get_node(p).area == node.area
  File "/.venv/lib/python3.8/site-packages/olivertwist/manifest.py", line 95, in area
    return self.data["fqn"][2]
IndexError: list index out of range

Latest pypi version is at 0.2 which does not include features such as add-rules-from

It would be good if adding rules was as simple as telling CLI to look in a directory containing .py scripts that look something like:

@Rule(link="http://example.com/my-rule")
def obey_my_rule():
   ...
   

Hi all,

I was exploring the package and I wasn't sure how stg and marts models were identified by the package until I had a chat with Paul. I assumed it was using string matching (or folder structure), but I think it would be good to add some notes about this to the docs here: https://github.com/autotraderuk/oliver-twist/blob/main/docs/rules.md#no-disabled-models

add the following rules:
lineage checks:

• 🆗 Disabled Scripts
• 🆗 Orphan models
• 🆗 Rejoined models. Given following model dependency: A -> B -> C. Model C should not depend on A
• 🆗 Staging scripts referencing a staging script in a different area
• 🆗 Staging scripts referencing a mart
• 🆗 Marts directly referencing a source

meta checks on physical models(model that is not ephemeral):

• 🆗 no owner defined as per(https://docs.getdbt.com/reference/resource-properties/meta/#designate-a-model-owner)
• no description defined
• no pii defined

advanced lineage check:

• detect similar sub graphs which they originate from the same set of sources to figure out if there is a potential code reuse(enforce single source of truth, thinking that it would highlight duplicate code)

Hiya!

I'm trying to update to dbt 0.20.0, and it seems dbt now requires Jinja2 2.11.3 (see here) and Olivertwist requires Jinja2 2.11.2 (see here).

Obviously this is causing an error for me.

Let me know if there is a better place or way to report issues like this. And also if you would need more information around set up to troubleshoot this.

Thank you in advance!

Version: olivertwist 0.2

See:
http://olivertwi.st/404.html

I'm thinking about using oliver-twist to help with auditing our dbt project and just wondering if this project is being actively maintained/accepting feature requests/etc?

Hello team,

We are mainly using Oliver twist in the CircleCi pipeline for DBT projects. Everytime OliverTwist is used we get this error:
Error: Invalid value for 'INPUT': Could not open file: manifest.json: No such file or directory

We tested OliverTwist locally (olivertwist check manifest.json) and we get the same error. Any idea how to resolve it?

Thank you!

Raed

Given we parse manifest.json into a networkx graph it feels sensible to just parse the pickled version output by DBT

https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows

Read rule configuration from a single config yml file

global:
    - id: <rule_name_1>
      enabled: false
    - id: <rule_name_2>
      enabled: false

All rules will be enabled be default.

The specified version of colorama (0.4.4) is not compatible with the latest dbt release (0.18.1) which depends on colorama (>=0.3.9,<0.4.4). This means you cannot have olivertwist & dbt within the same poetry project

To reproduce

$ poetry init --name example -n
$ poetry add "dbt=^0.18.1"
$ poetry add "olivertwist=^0.1.2"

Updating dependencies
Resolving dependencies... (0.3s)

[SolverProblemError]
Because no versions of olivertwist match >0.1.2,<0.2.0
 and olivertwist (0.1.2) depends on colorama (>=0.4.4,<0.5.0), olivertwist (>=0.1.2,<0.2.0) requires colorama (>=0.4.4,<0.5.0).
And because dbt-core (0.18.1) depends on colorama (>=0.3.9,<0.4.4), olivertwist (>=0.1.2,<0.2.0) is incompatible with dbt-core (0.18.1).
And because dbt (0.18.1) depends on dbt-core (0.18.1)
 and no versions of dbt match >0.18.1,<0.19.0, olivertwist (>=0.1.2,<0.2.0) is incompatible with dbt (>=0.18.1,<0.19.0).
So, because example depends on both dbt (^0.18.1) and olivertwist (^0.1.2), version solving failed.

From a quick look, could colorama be removed as a dependency entirely?

Enable users to load their own rules.

We want to write rules against our own metadata which wouldn't be generally reusable.

Right now rules appear to be applied to all models by default. An ability to configure a way to ignore a specific model would be really appreciated especially as one is adopting oliver twist it can sometimes require a significant refactor to get all the models in line with the auditing rules. Being able to specific specific models to be ignored would be quite useful. This probably should be doable either in the olivertwist.yml file or in the model.yml file in the meta field specifying some kind of audit ignore

Create a new docs folder to consolidate images and diagrams used in documentation

With configuration file, it is now possible to run with no rules enabled.
Currently this result in a success response from check.
It would be better to exit early with a warning that no enabled rules were loaded.

Currently, the configuration is applied globally. The rules and severity levels should be configurable against individual models.