fivetran / dbt_stripe_source Goto Github PK
View Code? Open in Web Editor NEWFivetran's Stripe source dbt package
Home Page: https://fivetran.github.io/dbt_stripe_source/
License: Apache License 2.0
Fivetran's Stripe source dbt package
Home Page: https://fivetran.github.io/dbt_stripe_source/
License: Apache License 2.0
Are you a Fivetran customer?
Meera, Consultant, Ephemeral Tattoos
Is your feature request related to a problem? Please describe.
I'm trying to use dbt-utils 0.7.0 (the latest release) so that I can use dbt 0.20.0 with my M1 mac, but this is conflicting with the dbt-utils version requirement in my Fivetran package
Describe the solution you'd like
Change the version check to allow for dbt-utils 0.7.0
Describe alternatives you've considered
Additional context
Please indicate the level of urgency and business impact of this request
Are you interested in contributing to this package?
The below columns to be added
STATUS_TRANSITIONS_FINALIZED_AT
STATUS_TRANSITIONS_PAID_AT
STATUS_TRANSITIONS_VOIDED_AT
STATUS_TRANSITIONS_MARKED_UNCOLLECTIBLE_AT
None
No response
I want to disable the below tables from processing. Can you please let me know how to do it ?
dbt_project .yml below
Internal - Add databricks compatibility
No response
No response
Hi there, I would like to run this package on our stripe prod data aswell as the test data (using the fivetran test connector). I can override schema names accordingly. But test customers will never be pulled in because their livemode flag will be set to false. Would it be worth adding some configuration option to enable to run this package on test data?
Just wondering if there are some considerations that I'm not aware of?
many thanks,
Christoph.
The latest release of the fivetran_utils package allows us to instinctively infer if the source table the package is using is in fact present, if it is not the package will create an empty table to ensure the downstream transformations succeed.
However, the Stripe package is not currently compatible with this feature due to the livemode_predicate which exists in the *_tmp
staging models. In order for the new empty data model creation to be successful in the Stripe package, we will need to move this predicate macro to the non *_tmp
staging model.
Not leveraging the new empty model creation feature within the tmp models.
No response
An error in my Fivetran/Stripe package today with uniqueness in stg_stripe_invoice_line_items
model.
Seems to be testing for uniqueness on unique_invoice_line_item_id
but I have a few records where it is not unique through the data source.
17:49:29 3 of 3 START test unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id [RUN]
17:49:30 3 of 3 FAIL 3 unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id . [FAIL 3 in 1.09s]
As this stg_stripe_invoice_line_items
has 2 PK's I expect the unique_stg_stripe__invoice_line_item_unique_invoice_line_item_id
to not exist.
dbt_utils_unique_combination_of_columns_stg_stripe__invoice_line_item_invoice_line_item_id__invoice_id__source_relation
is already there to ensure the combo is unique
#### Variables for Fivetran/stripe package ####
# stripe_database: database
stripe_schema: source_ft_stripe
stripe__using_invoices: True #Disable if you are not using the invoice and invoice_line_item tables
stripe__using_payment_method: False #Disable if you are not using the payment_method and payment_method_card tables
stripe__using_subscriptions: True #Disable if you are not using the subscription and plan/price tables.
stripe__using_credit_notes: False #Enable if you are using the credit note tables.
- package: fivetran/stripe
version: [">=0.12.0", "<0.13.0"]
bigquery
18:16:55 Running with dbt=1.6.11
I think you just need to remove this test?
Field renaming should take place in staging models (ex, the in the account staging model, rename id to account_id
and name to account_name
). We've done this in a few places but not all.
However note this will have downstream implications as most of the renaming occurs in the transforms package.
No response
No response
Are you a Fivetran customer?
Benoit Mayer, Stedi Inc
Is your feature request related to a problem? Please describe.
We store customer metadata in Stripe on the customers and would like it to flow through to reporting. the metadata is essential to our reporting.
Describe the solution you'd like
add metadata on customers as implemented for transactions as described https://github.com/fivetran/dbt_stripe_source/tree/v0.4.3#pivoting-out-metadata-properties
Describe alternatives you've considered
Please indicate the level of urgency and business impact of this request
not critical but time sensitive. by end of month would be ideal
Are you interested in contributing to this package?
Hi there,
In the stg_stripe__plan
model, any reason we need this bit?
Here's the direct link to the code
{% if target.type == 'bigquery' %}
'interval' as plan_interval,
{% else %}
interval as plan_interval,
{% endif %}
The issue with the above is I no longer have ability to see the ACTUAL interval of the plan. Instead it's just a blanket interval
constant, making plan_interval
useless.
Is there a particular issue this only happens for BigQuery
?
Are you a Fivetran customer?
Yes. Ramiz Bozai, Senior Data Analyst at Cerebral
Is your feature request related to a problem? Please describe.
It would be great to have an identifier denoting which data/records came from which source as I union together our different stripe tables from different connections.
Describe the solution you'd like
Having a column indicating what the source of the data is (like country) would allow us to filter that data based on the column.
Describe alternatives you've considered
This is more of a nice to have, but I understand I can possible fork this package and potentially try to add this myself.
Additional context
N/A
Please indicate the level of urgency and business impact of this request
Not urgent. Just a nice to have feature.
Are you interested in contributing to this package?
As identified within the dbt_stripe Issue #45, it seems that the fal
target.type is not supported within conditionals found throughout the package. In particular, the get_plan_columns conditional is exclusively looking for the target.type in BigQuery, Spark, or Databricks. However, if a user has the fal adapter, then it will fail as the quoting is needed but the adapter is not included in the conditional.
This feature is to explore supporting the fal
adapter and how it differs from other warehouses.
Do not use the fal adapter.
No response
Is is possible to provide some feedback on this staging model and the exclusion of invoice line items starting with sub_
? Trying to find out in Stripe's documentation, I wasn't able to find something relevant. Also according to Stripe:
There are actually 3 types of ID that can appear in the invoice_line_item. What type of ID appears depends on where the item has been added.
sub_ - The ID associated with a subscription on a customer object.
si_ - The ID associated with a subscription item. Typically used for multiplan subscriptions, where the plan becomes the subscription item.
sli_ - The ID associated with a subscription line item. It essentially defines a subscription item on an invoice.
So, it seems that invoice line items starting with sub_
are related to subscriptions, so in stripe__subscription_line_items
model of dbt_stripe
those are excluded even thought it seems like they shouldn't.
The current definition of the created
field within the balance_transaction
source should explicitly define the timezone the timestamp is reported within. The stripe connector syncs this field as UTC and we should address it as such.
N/A
The field definition details the UTC timezone.
N/A
packages:
bigquery
v 1.1.0
No response
The new model for subscription item to be added
No response
No response
The model is still using stg_stripe__subscription. This is throwing error as the table doesn't not exist . It has been changed to stg_stripe__subscription_history
STRIPE.SUBSCRIPTION' does not exist or not authorized.
The model is expected to run fine
stripe_database: FIVETRAN_DB
stripe_schema: STRIPE
0.7.2
snowflake
1.1.1
No response
We should leverage the adapter dispatch for all macros in order to allow customers to override them when needed. (livemode_predicate
)
No response
We can test that the macro can be overridden using the dispatch config.
na
na
other (mention it in "Additional Context")
na
No response
Are you a Fivetran customer?
Original request identified by @sachams
Is your feature request related to a problem? Please describe.
No, this is to add the metadata
fields to staging models.
Describe the solution you'd like
Ideally a customer would be able to add the metadata field and possible parse the field out as well.
Current PR Work
PR #18
Remove this filter and persist the is_deleted
field so that deleted customers continue downstream (and we can tell that they're deleted).
this will provide more complete data to places where we join in customer data (like here) and will allow for better matching in the current Customer360 project.
note: this will increase the # of rows in the stripe__customer_overview
table, but likely not by a huge amount.
also this will not require a change to the transform package, as the is_deleted
field will be automatically picked up by this
No response
No response
Are you a current Fivetran customer?
Yes
Describe the bug
Found 5k rows in my raw Payment Intents table that are missing from the staging Payment Intents table. Realized they were in the tmp view, meaning that I hadn't re-run dbt.
Steps to reproduce
select count(*) from raw_data.stripe.payment_intent raw
left join analytics_data.dbt_meera.stg_stripe__payment_intent stg on raw.id = stg.payment_intent_id
where stg.payment_intent_id is null
Expected behavior
I'd expect no missing rows across my raw data and staging data
Project variables configuration
version: '1.0'
require-dbt-version: ">=0.14.0"
source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]
target-path: "target"
clean-targets:
- "target"
- "dbt_modules"
quoting:
database: false
identifier: false
schema: false
models:
transient: true
stripe_source:
+schema: # Left blank so that will write to schema "dbt"
on-run-end:
- "{{ grant_select_on_schemas(schemas, 'reporter') }}"
- "{{ grant_select_on_schemas(schemas, 'transformer') }}"
config-version: 2
vars:
# Defined variables for Fivetran Stripe package
stripe_schema: stripe
stripe_database: raw_data
using_invoices: False #Disabled since we are not using the invoice and invoice_line_item tables
using_subscriptions: False #Disabled since we are not using the subscription and plan tables.
Package Version
packages:
- package: fishtown-analytics/dbt_utils
version: 0.6.5
- package: fivetran/stripe_source
version: 0.3.1
Warehouse
Hi,
I noticed in the docs you only have a single stripe database, is it possible to have multiple Stripe databases? For example to support multicurrency and not be charged transaction fees, you have to have multiple Stripe accounts.
So we have like:
stripe_foo
stripe_bar
stripe_baz
as our database names.
It would be beneficial to add the subscription_id
field to the invoice staging table. This field could then be leveraged in downstream transformations.
Fork the package and leverage this field or use the source directly.
No response
Missing columns
NA
Can this be expedited ? This model will not be useful to me as they are important columns
Are you a Fivetran customer?
Fivetran created FR
Is your feature request related to a problem? Please describe.
This feature request is not related to a problem. However, I could see this being an issue if certain customers have history mode enabled and want the package to run for just the latest records of the orders
and subscriptions
source tables.
Describe the solution you'd like
The Stripe release notes for February 2022 indicate that there are two new tables ORDER_HISTORY
and SUBSCRIPTION_HISTORY
. This FR is to include those two tables into the package and allow users to disable the history mode and only select the latest records.
Describe alternatives you've considered
Using the non-history ORDER
and SUBSCRIPTION
source tables.
The test dbt_utils_unique_combination_of_columns_stg_stripe__fee_balance_transaction_id__source_relation
is defined as:
- name: stg_stripe__fee
description: The details of a fee associated with a balance transaction
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- balance_transaction_id
- source_relation
However, the Fee table has an additional column index
that is part of the composite primary key as described here:
So this test is giving false positives
No response
The expected behavior would be for this test to not give false positives and include the index column as part of the unique column set.
name: 'chabi'
version: '1.0.0'
config-version: 2
profile: "{{ env_var('DBT_PROFILE', '') }}"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
- "target"
- "dbt_packages"
-
models:
stripe:
+schema:
intermediate:
+schema: stg
stripe_source:
+schema: stg
vars:
stripe_schema: stripe
stripe_timezone: Australia/Sydney
stripe__using_credit_notes: true
packages:
- package: dbt-labs/dbt_utils
version: [">=1.0.0", "<2.0.0"]
- package: dbt-labs/codegen
version: 0.9.0
- package: fivetran/stripe
version: 0.9.0
snowflake
Core:
- installed: 1.4.1
- latest: 1.4.5 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- snowflake: 1.4.0 - Update available!
At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
No response
We include the phone
field in the get_customer_columns macro, but we do not include it in the staging model. We only include the shipping_phone
but i believe both will be needed for our identity resolution project
just use shipping_phone, but shipping_phone is sometimes just the company phone, while phone
is the individual's
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.