ltchikindas / dbt-utils Goto Github PK

This dbt package contains macros that can be (re)used across dbt projects.

This macro returns the current timestamp.

Usage:

{{ dbt_utils.current_timestamp() }}

This macro adds a time/day interval to the supplied date/timestamp. Note: The datepart argument is database-specific.

Usage:

{{ dbt_utils.dateadd(datepart='day', interval=1, from_date_or_timestamp='2017-01-01') }}

This macro calculates the difference between two dates.

Usage:

{{ dbt_utils.datediff("'2018-01-01'", "'2018-01-20'", 'day') }}

This macro splits a string of text using the supplied delimiter and returns the supplied part number (1-indexed).

Usage:

{{ dbt_utils.split_part(string_text='1,2,3', delimiter_text=',', part_number=1) }}

Truncates a date or timestamp to the specified datepart. Note: The datepart argument is database-specific.

Usage:

{{ dbt_utils.date_trunc(datepart, date) }}

Gets the last day for a given date and datepart. Notes:

• The datepart argument is database-specific.
• This macro currently only supports dateparts of month and quarter.

Usage:

{{ dbt_utils.last_day(date, datepart) }}

This macro returns the sql required to build a date spine.

Usage:

{{ dbt_utils.date_spine(
    datepart="minute",
    start_date="to_date('01/01/2016', 'mm/dd/yyyy')",
    end_date="dateadd(week, 1, current_date)"
   )
}}

This macro calculates the haversine distance between a pair of x/y coordinates.

Usage:

{{ dbt_utils.haversine_distance(lat1=<float>,lon1=<float>,lat2=<float>,lon2=<float>) }}

This schema test asserts the equality of two relations.

Usage:

model_name:
  constraints:
    dbt_utils.equality:
      - ref('other_table_name')

This schema test asserts that there is data in the referenced model at least as recent as the defined interval prior to the current timestamp.

Usage:

model_name:
    constraints:
        dbt_utils.recency:
            - {field: created_at, datepart: day, interval: 1}

This schema test asserts if column has at least one value.

Usage:

model_name:
  constraints:
    dbt_utils.at_least_one:
      - column_name

This schema test asserts if column does not have same value in all rows.

Usage:

model_name:
  constraints:
    dbt_utils.not_constant:
      - column_name

This schema test asserts if values in a given column have exactly the same cardinality as values from a different column in a different model.

Usage:

model_name:
  constraints:
    dbt_utils.cardinality_equality:
    - {from: column_name, to: ref('other_model_name'), field: other_column_name}

This macro returns the unique values for a column in a given table.

Usage:

-- Returns a list of the top 50 states in the `users` table
{% set states = dbt_utils.get_column_values(table=ref('users'), column='state', max_records=50) %}

{% for state in states %}
    ...
{% endfor %}

...

This macro returns a list of tables that match a given prefix, with an optional exclusion pattern. It's particularly handy paired with union_tables.

Usage:

-- Returns a list of tables that match schema.prefix%
{{ set tables = dbt_utils.get_tables_by_prefix('schema', 'prefix')}}

-- Returns a list of tables as above, excluding any with underscores
{{ set tables = dbt_utils.get_tables_by_prefix('schema', 'prefix', '%_%')}}

This macro build a group by statement for fields 1...N

Usage:

{{ dbt_utils.group_by(n=3) }} --> group by 1,2,3

This macro generates a list of all fields that exist in the from relation, excluding any fields listed in the except argument. The construction is identical to select * from {{ref('my_model')}}, replacing star (*) with the star macro.

Usage:

select
{{ dbt_utils.star(from=ref('my_model'), except=["exclude_field_1", "exclude_field_2"]) }}
from {{ref('my_model')}}

This macro implements an "outer union." The list of tables provided to this macro will be unioned together, and any columns exclusive to a subset of these tables will be filled with null where not present. The column_override argument is used to explicitly assign the column type for a set of columns.

Usage:

{{ dbt_utils.union_tables(
    tables=[ref('table_1'), ref('table_2')], 
    column_override={"some_field": "varchar(100)"}, 
    exclude=["some_other_field"]
) }}

This macro implements a cross-database mechanism to generate an arbitrarily long list of numbers. Specify the maximum number you'd like in your list and it will create a 1-indexed SQL result set.

Usage:

{{ dbt_utils.generate_series(upper_bound=1000) }}

Implements a cross-database way to generate a hashed surrogate key using the fields specified.

Usage:

{{ dbt_utils.surrogate_key('field_a', 'field_b'[,...]) }}

This macro pivots values from rows to columns.

Usage:

{{ dbt_utils.pivot(<column>, <list of values>) }}

Example:

Input: public.test

| size | color |
|------|-------|
| S    | red   |
| S    | blue  |
| S    | red   |
| M    | red   |

select
  size,
  {{ dbt_utils.pivot('color', dbt_utils.get_column_values('public.test',
                                                         'color')) }}
from public.test
group by size

Output:

| size | red | blue |
|------|-----|------|
| S    | 2   | 1    |
| M    | 1   | 0    |

Arguments:

- column: Column name, required
- values: List of row values to turn into columns, required
- excluded_values: list of values you do not want pivoted. This should be 
                 passed in parentheses ie excluded_values=('foo','bar'). 
                 default is empty
- alias: Whether to create column aliases, default is True
- coalesce: Whether the whole statement should coalesce to 0 
          ie coalesce(agg(case),0), default is False
- agg: SQL aggregation function, default is sum
- total: Whether all values (except excluded) should be summed for 
       a total column, default is False
- distinct: Whether vals in the agg should be distinct ie count(distinct),
       default is False
- cmp: SQL value comparison, default is =
- prefix: Column alias prefix, default is blank
- suffix: Column alias postfix, default is blank
- then_value: Value to use if comparison succeeds, default is 1
- else_value: Value to use if comparison fails, default is null

This macro extracts a url parameter from a column containing a url.

Usage:

{{ dbt_utils.get_url_parameter(field='page_url', url_parameter='utm_source') }}

insert_by_period allows dbt to insert records into a table one period (i.e. day, week) at a time.

This materialization is appropriate for event data that can be processed in discrete periods. It is similar in concept to the built-in incremental materialization, but has the added benefit of building the model in chunks even during a full-refresh so is particularly useful for models where the initial run can be problematic.

Should a run of a model using this materialization be interrupted, a subsequent run will continue building the target table from where it was interrupted (granted the --full-refresh flag is omitted).

Progress is logged in the command line for easy monitoring.

Usage:

{{
  config(
    materialized = "insert_by_period",
    period = "day",
    timestamp_field = "created_at",
    start_date = "2018-01-01",
    stop_date = "2018-06-01"
}}

with events as (

  select *
  from {{ ref('events') }}
  where __PERIOD_FILTER__ -- This will be replaced with a filter in the materialization code

)

....complex aggregates here....

Configuration values:

• period: period to break the model into, must be a valid datepart (default='Week')
• timestamp_field: the column name of the timestamp field that will be used to break the model into smaller queries
• start_date: literal date or timestamp - generally choose a date that is earlier than the start of your data
• stop_date: literal date or timestamp (default=current_timestamp)

Caveats:

• This materialization is compatible with dbt 0.10.1.
• This materialization has been written for Redshift.
• This materialization can only be used for a model where records are not expected to change after they are created.
• Any model post-hooks that use {{ this }} will fail using this materialization. For example:

models:
    project-name:
        post-hook: "grant select on {{ this }} to db_reader"

A useful workaround is to change the above post-hook to:

        post-hook: "grant select on {{ this.schema }}.{{ this.name }} to db_reader"

We welcome contributions to this repo! To contribute a new feature or a fix, please open a Pull Request with 1) your changes, 2) updated documentation for the README.md file, and 3) a working integration test. See this page for more information.

• What is dbt?
• Read the dbt viewpoint
• Installation
• Join the chat on Slack for live questions and support.

Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.