palmetto / palm-dbt Goto Github PK

Context
Our test coverage on this plugin is not great. It would be great to take the time to test as much of the code as we can.

Is your feature request related to a problem? Please describe.
No, this is an improvement that will help keep the codebase healthy in the long-run.

Describe the solution you'd like
Implement tests for the following (in loose priority order):

• dbt_plugin.py + plugins/dbt/init.py
• dbt_palm_utils.py
• sql_to_dbt_.py
• local_user_lookup.py
• Commands (this may be very tricky, take a shot but consider breaking this out to a separate issue if there are significant troubles writing meaningful tests around these)

Describe alternatives you've considered
Don't test the code, watch the project crumble over time.

Additional context
We are using pytest, there are already some tests around dbt_containerizer, look at palm-cli test_cli.py for reference in re. testing commands.

Is there an existing feature request for this?

• I have searched the existing issues

Subject of the issue

The test branch name is hardcoded into the macro. Make it parameterized so that others can configure it for their specific setups.

example: {%- if target.database != "TEST" -%}

Your environment

• Version of palm
• Operating system
• Docker / Docker compose version
• Relevant dependencies and versions

Steps to reproduce

Try testing with a database that isn't named "TEST"

Expected behaviour

Pass name of test database in the macro as a param

Actual behaviour

"TEST" is hardcoded in the macro

Describe the bug

• palm models hard coded 'arbor' on line 55 and 76
• palm sql_to_dbt as hard coded 'arbor' on line 18

Your environment

• palm version: 2.1.0
• OS: Mac

How to reproduce
palm model new --name "test_dim" --model-type dim

Expected behavior
palm model should generate director/files based on the template

Actual behavior
FileNotFoundError: [Errno 2] No such file or directory: '.../pdp-arbor/models/arbor/dim/dim_test_dim'

Additional context

• Remove references to 'Arbor'

Context
When containerizing dbt, I want the ability to select the dbt version I am using

Is your feature request related to a problem? Please describe.
When I run containerize, it defaults to 0.21.0. I wouldn't catch this unless I knew to double-check the docker file.

Describe the solution you'd like
Prompt the user to enter (or at least confirm) the dbt version (if the package is getting it from somewhere else).

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the bug
As a palm-dbt user, I expect the latest version of palm-dbt will install the macros I need to run the dbt projects. However, when I attempt to install them I get the error below.

Your environment

• palm version: 2.1.0
• palm-dbt version: 0.0.4
• OS:
• Docker version:
• Other dependencies & versions:

How to reproduce

python3 -m pip uninstall palm || true
python3 -m pip uninstall palm-dbt || true
pip install palm
pip install palm-dbt
cd jaffle_shop
palm init -p dbt

Expected behavior
palm install installs macros

Actual behavior

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/palm", line 8, in <module>
    sys.exit(cli())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/palm/plugins/dbt/commands/cmd_install.py", line 39, in cli
    shutil.copy(Path(macro_template_path, macro), Path(macros_path, macro))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 415, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/palm/plugins/dbt/macros/drop_branch_schemas.sql'

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Context
dbt just released v1.0! We want to support the latest version

Is your feature request related to a problem? Please describe.
Improvement - support the latest and greatest!

Describe the solution you'd like

• Update containerize version support to include v1.0
• Implement any necessary changes/improvements to make the most of dbt v1.0!

Describe alternatives you've considered

• There is no alternative, we need to do this!

Is there an existing feature request for this?

• I have searched the existing issues

what is the general idea or problem you want to solve?

currently the stock base image created by palm init does not contain deps. most of the commands like run, test, cycle etc will execute deps first, so you always have fresh deps (though this does slow them down significantly). If we move deps to the build cycle, what are the pros? does this make sense?

Desired Implementation

run dbt deps in the image build. Then move deps to a stand-alone palm command and a flag for the other commands.

Risks

this could leave you working with outdated deps, and when a new container is built the behavior might be unexpected. Especially if dbt packages are poorly pinned.

Context
What are you trying to do and how would you want to do it differently? Is it something you currently cannot do?

Use palm-dbt with dbt 1.0.0

Is your feature request related to a problem? Please describe.
When using palm-dbt with dbt 1.0.0, docker requires build args for username and password, not just env vars

Describe the solution you'd like
A clear and concise description of what you want to happen.

Update palm-dbt and containerize command to support dbt 1.0.0

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is there an existing feature request for this?

• I have searched the existing issues

Describe the bug
I ran palm run for the first time in a new project repo, so the plugin config walk-through ran. The prompt asks if you have prod artifacts stored locally, and I answered "N" and chose the default "target" folder for local artifacts. Right after this, an error threw on the dbtPluginConfigModel:

Invalid config for plugin dbt: 1 validation error for dbtPluginConfigModel
dbt_artifacts_prod
  Field required [type=missing, input_value={'dbt_artifacts_local': 'target'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.1/v/missing

Your environment

• palm version: 2.5.3
• OS: Mac OS
• Docker version: 24.0.2
• Other dependencies & versions:

How to reproduce
In a new dbt project, run any palm command or kick off the pluginconfig initialization and click "No" for the prompt about prod artifacts stored locally.

Expected behavior
The flow handles the answer no and simply doesn't throw an error if you have no prod artifacts stored locally.

Actual behavior
The flow threw an error and quit.

Screenshots
None

Additional context
N/A

Context
palm containerize doesn't support dbt v1.4 or v1.5

Is your feature request related to a problem? Please describe.
Any dbt version that can run with palm should be supported by the containerizer.

Describe the solution you'd like
palm containerize should not bork when inputs greater than 1.3 are provided in the walk-through. We should increase the max version for containerize to include the more recent dbt minor versions, 1.4 and 1.5 (and maybe the upcoming 1.6 depending on when this gets done).

Describe alternatives you've considered
N/A

Additional context

Is there an existing feature request for this?

• I have searched the existing issues

Describe the bug
dbt 1.5 has renamed the environment variables for defer and state artifact usage. This means palm throws warnings on every dbt command that utilizes state or run_results

Your environment

• palm version: 2.5.3
• OS: Mac OS
• Docker version:
• Other dependencies & versions: dbt V1.5.2

How to reproduce
Upgrade to dbt 1.5+ and run palm

Expected behavior
Palm doesn't rely on deprecated features and runs without verbose warnings

Actual behavior
Every palm command run produces warnings from dbt

Screenshots
None

Additional context
N/A

Context
dbt now has 5 minor versions in V1 so palm should support all five and deprecate support below V1.

Is your feature request related to a problem? Please describe.
Palm rejects the two newest dbt minor versions, which it should support. Also, <= V0.19 is not supported by dbt any longer, so it doesn't need to be supported by palm either.

Describe the solution you'd like
Deprecate v0.19 and add support for dbt v1.4 and v1.5

Describe alternatives you've considered
N/A

Additional context

Is there an existing feature request for this?

• I have searched the existing issues

Describe the bug
The behavior of dbt run on model build failure is to exit with a non-zero status code. The subprocess call raises an error when ctx.obj.run_in_docker receives the exit code 1 from dbt. Thus, for all of the commands that need to run in succession (dbt run && dbt run-operation drop_branch_schemas) the failure is preventing the full set of commands from being run. In development, this is troublesome because failed builds are unintentionally being persisted in the target. We want idempotent runs, and we don't have them unless all models build perfectly the first time (unlikely...).

Your environment

• palm version: 2.0.1
• OS: Mac OS Big Sur
• Docker version: 20.10.8
• Other dependencies & versions: palm-dbt: 0.0.2

How to reproduce
Misspell a column name in a dbt model in the middle of a dag which is used by downstream models. Execute palm run.

Expected behavior
I expect a failed dbt run to complete cleanly. If models fail to build, then the test cycle in palm cycle should probably be skipped, but the cleanup step which runs run-operation drop_branch_schemas should still complete unless the --persist flag was provided.

Actual behavior
After your Database error in model.... message from dbt on a failed run, you get a large stacktrace, and dbt run-operation drop_branch_schemas never gets run.

Screenshots
...

Additional context
...