Light

yu-iskw / dbt-airflow-macros Goto Github PK

View Code? Open in Web Editor NEW

13.0 3.0 2.0 45 KB

Dbt package for Apache Airflow inspired macros

Home Page: https://hub.getdbt.com/yu-iskw/dbt_airflow_macros/latest/

License: Apache License 2.0

Shell 93.47% Makefile 6.53%

dbt dbt-packages airflow

dbt-airflow-macros's Introduction

dbt-airflow-macros

This is a dbt package to use macros inspired by Apache Airflow. The macros are not warehouse-specific. Those work with any warehouse.

Recommended environments
Installation
Environment variable
Macros
Examples
Tested environments
Links

Recommended environments

Python: 3.6, 3.7 and 3.8
dbt: >=0.17.0

Installation

Include in packages.yaml

packages:
  - git: "https://github.com/yu-iskw/dbt-airflow-macros.git"
    revision: 0.2.2

Environment variable

The shell environment variable EXECUTION_DATE enables us to pass the date and time for the dbt macros. The ISO 8601 format is available, because the package uses datetime.datetime.fromisoformat internally. If we don't set EXECUTION_DATE, then it is set to the current UTC date and time.

EXECUTION_DATE="2020-01-01T01:23:45" dbt run

EXECUTION_DATE="2020-01-01" dbt run

Macros

The package includes the following macros. timezone is an optional argument. When it is set to none, the timezone is UTC by default.

Macro	Description
`dbt_airflow_macros.execution_date(timezone=none)`	`datetime.datetime` object
`dbt_airflow_macros.ds(timezone=none)`	`"%Y-%m-%d"` of execution time
`dbt_airflow_macros.ds_nodash(timezone=none)`	`"%Y%m%d"` of execution time
`dbt_airflow_macros.ts(timezone=none)`	`"%Y-%m-%dT%H:%M:%S+00:00"` of execution time
`dbt_airflow_macros.ts_nodash(timezone=none)`	`"%Y%m%dT%H%M%S"`of execution_time

Examples

test_macros.sql is a file to test the macros on PostgreSQL. Those would be helpful to understand how to use the macros.

Tested environments

The continuous integration tested the dbt package with the environments below.

dbt-airflow-macros==0.2.x
- Python: 3.6, 3.7 and 3.8
- dbt: 0.17.2, 0.18.1, 0.19.0, 0.20.0 and 0.21.0

Links

dbt-airflow-macros's People

Contributors

Stargazers

Watchers

Forkers

hoangvu-ct lhvubtqn

dbt-airflow-macros's Issues

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/yu-iskw/dbt-airflow-macros/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Security Policy violation Branch Protection

This issue was automatically created by Allstar.

Security Policy Violation
Dismiss stale reviews not configured for branch main

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Airflow execution_date is not the same as dbt_airflow_macros.execution_date()

I notice that the dbt_airflow_macros.execution_date() only generates values for when the dbt cloud job is triggered from airflow and not when the dag is ran.

I will like to be able to import the execution_date() of airflow task run to dbt. Do you know any way to do this?

Add integration tests

Goals

Integration tests will be implemented.

Update environment variable name or add option to set it

In my Airflow environment, the environment variable name for execution date is AIRFLOW_CTX_EXECUTION_DATE. It wasn't easy for me to set another environment variable to be able to use this package. Can the environment variable name be updated or at least add an option to override the default?

Register the package in dbt hub

Goals

The package will be registered in dbt hub.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.