fill_staging_columns not applying type about dbt_fivetran_utils HOT 2 CLOSED

Hi @leinemann thanks for posting this question. Hopefully I can help address all of these questions.

referring to this warning: https://github.com/fivetran/dbt_fivetran_utils#%EF%B8%8F-warning-%EF%B8%8F - am I understanding correctly that master is deprecated but new versions are still released? I can see recent release of 0.4.7 - what branch is used for latest development (as it doesn't seem to be the default branch 'master')?

• Yes that is correct, we have deprecated the master branch and are currently using releases/v0.4.latest as the base for all v0.4.x releases. We are hoping to adjust the default branch soon; however, some users are still using older version of our packages which rely on the default branch being master. Therefore, we cannot update the default branch until those users have migrated and is the result of the releases/v0.4.latest not being the default currently. We hope to update this soon.

specifically - how is this macro going to apply the cast into the given/desired datatype:
dbt_fivetran_utils/macros/fill_staging_columns.sql
{{ fivetran_utils.quote_column(column) }} as
from my understanding it's missing the cast.

• This macro does not intend to cast fields when filling them into the staging model. The casting only should occur if the staging column is not found in the source columns. You can see that casting occur here. We are setting the nonexistent field to null and then casting it to the appropriate datatype. If the field does exist, we do not intend to cast it. That type of intentional casting will take place in the staging model itself.

what is a replacement for the 'generate_columns_macro' macro and why is it discontinued. I.e. the latest version 4.7 doesn't seem to have: https://github.com/fivetran/dbt_fivetran_utils#generate_columns_macro-source but the documentation of fill_staging_columns still references it: https://github.com/fivetran/dbt_fivetran_utils/tree/v0.4.7#fill_staging_columns-source

• Thanks for calling out the discrepancy in the documentation. the generate_columns_macro does still exist, but we have moved it to a new repo. You can see the automation macros for our package development within the dbt_package_automations repo. We moved these automation macros here as we had a few occurrences where these macros caused compilation errors for some users when installing the fivetran utils package. Therefore, we decided to move them to a separate repo to ensure this couldn't occur, but our team could still leverage them in model development.

what is the current best practice to handle typing of staging columns?

• When developing new staging models, my team uses the staging_models_automation macro and bash commands to generate the specified macros and staging models. We then go into the models and clean up or specifically cast and rename fields in the staging models as necessary.
• As a note, this works for our team when needing to create staging models at scale. This may not be the best method for all use cases.

from dbt_fivetran_utils.

thank you so much for these extensive explanations @fivetran-joemarkiewicz , it all makes sense!

from dbt_fivetran_utils.