Code Monkey home page Code Monkey logo

dbt_xero's Introduction

Xero Transformation dbt Package (Docs)

πŸ“£ What does this dbt package do?

  • Produces modeled tables that leverage Xero data from Fivetran's connector in the format described by this ERD and builds off the output of our Xero source package.

  • Provides analytics-ready models, including a profit and loss report, general ledger, and balance sheet report.

  • Generates a comprehensive data dictionary of your source and modeled Xero data through the dbt docs site.

The following table provides a detailed list of all models materialized within this package by default.

Model Description
xero__general_ledger Each record represents a journal line item. Use the ledger to create the balance sheet and the profit and loss statement.
xero__profit_and_loss_report Each record represents a profit and loss line item at the month and account level.
xero__balance_sheet_report Each record represents the state of the balance sheet for a given account on a given month.
xero__invoice_line_items Each record represents an invoice line item enriched with the account, contact, and invoice information.

🎯 How do I use the dbt package?

Step 1: Prerequisites

To use this dbt package, you must have the following:

  • At least one Fivetran Xero connector syncing data into your destination.
  • A BigQuery, Snowflake, Redshift, PostgreSQL, or Databricks destination.

Step 2: Install the package

Include the following xero package version in your packages.yml file:

TIP: Check dbt Hub for the latest installation instructions or read the dbt docs for more information on installing packages.

packages:
  - package: fivetran/xero
    version: [">=0.6.0", "<0.7.0"] # we recommend using ranges to capture non-breaking changes automatically

Do NOT include the xero_source package in this file. The transformation package itself has a dependency on it and will install the source package as well.

Step 3: Define database and schema variables

By default, this package runs using your destination and the xero schema. If this is not where your Xero data is (for example, if your Xero schema is named xero_fivetran), add the following configuration to your root dbt_project.yml file:

vars:
    xero_schema: your_schema_name
    xero_database: your_database_name 

(Optional) Step 4: Additional configurations

Expand for configurations

Note about currency gains

If you are using multi-currency accounting in Xero, you are likely to have unrealized currency gains as part of your profit and loss statement. These gains/losses do not exist within the actual journals in Xero. As a result, you will find that those lines are missing from the outputs of this package. All realised currency gains will be present and your balance sheet will still balance.

Unioning Multiple Xero Connectors

If you have multiple Xero connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the source_relation column of each model. To use this functionality, you will need to set either (note that you cannot use both) the union_schemas or union_databases variables:

# dbt_project.yml
...
config-version: 2
vars:
  xero_source:
    union_schemas: ['xero_us','xero_ca'] # use this if the data is in different schemas/datasets of the same database/project
    union_databases: ['xero_us','xero_ca'] # use this if the data is in different databases/projects but uses the same schema name

Disabling and Enabling Models

When setting up your Xero connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Xero or have actively decided to not sync some tables. In order to disable the relevant functionality in the package, you will need to add the relevant variables.

By default, all variables are assumed to be true. You only need to add variables for the tables you would like to disable:

# dbt_project.yml

config-version: 2

vars:
    xero__using_credit_note: false                      # default is true
    xero__using_bank_transaction: false                 # default is true

For additional configurations for the source models, visit the Xero source package.

Changing the Build Schema

By default this package will build the Xero Source staging models within a schema titled (<target_schema> + _stg_xero) and the Xero final transform models within a schema titled (<target_schema> + _xero) in your target database. To overwrite this behavior, add the following configuration to your dbt_project.yml file:

# dbt_project.yml

...
models:
    xero:
        +schema: my_new_final_models_schema # leave blank for just the target_schema
    xero_source:
        +schema: my_new_staging_models_schema # leave blank for just the target_schema

Change the source table references

If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:

IMPORTANT: See this project's dbt_project.yml variable declarations to see the expected names.

vars:
    xero_<default_source_table_name>_identifier: your_table_name 

(Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Coreβ„’

Expand for more details

Fivetran offers the ability for you to orchestrate your dbt project through Fivetran Transformations for dbt Coreβ„’. Learn how to set up your project for orchestration through Fivetran in our Transformations for dbt Core setup guides.

πŸ” Does this package have dependencies?

This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the dbt hub site.

IMPORTANT: If you have any of these dependent packages in your own packages.yml file, we highly recommend that you remove them from your root packages.yml to avoid package version conflicts.

packages:
    - package: fivetran/dbt_xero_source
      version: [">=0.6.0", "<0.7.0"]

    - package: fivetran/fivetran_utils
      version: [">=0.4.0", "<0.5.0"]

    - package: dbt-labs/dbt_utils
      version: [">=1.0.0", "<2.0.0"]

πŸ™Œ How is this package maintained and can I contribute?

Package Maintenance

The Fivetran team maintaining this package only maintains the latest version of the package. We highly recommend you stay consistent with the latest version of the package and refer to the CHANGELOG and release notes for more information on changes across versions.

Contributions

A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions!

We highly encourage and welcome contributions to this package. Check out this dbt Discourse article on the best workflow for contributing to a package!

πŸͺ Are there any resources available?

  • If you have questions or want to reach out for help, please refer to the GitHub Issue section to find the right avenue of support for you.
  • If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our Feedback Form.
  • Have questions or want to just say hi? Book a time during our office hours on Calendly or email us at [email protected].

dbt_xero's People

Contributors

danieltaft avatar dylanbaker avatar fivetran-abhijeet avatar fivetran-catfritz avatar fivetran-dejantucakov avatar fivetran-jamie avatar fivetran-joemarkiewicz avatar fivetran-reneeli avatar fivetran-sheringuyen avatar jsagasta avatar kristin-bagnall avatar sjmunoz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbt_xero's Issues

BUG - rows in the xero__general_ledger model that have a NULL value in the journal_line_id column

Are you a current Fivetran customer?

Yes - Jared - Samarkand Global - Data Team Lead

Describe the bug

There are instances where there are rows in the xero__general_ledger model that have a NULL value in the journal_line_id column.

This causes the test:

dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation

to fail because the NULL values are seen to be the same journal_line_id

Expected behavior

I'm not 100%

  • Either something is wrong with our data and / or the Fivetran connector because I would expect every journal line to have an ID.
  • Or update the test to include the journal_id or journal_number. But this might not work if a single journal has more than one row that have null values in journal_line_id
  • Add a "where journal_line_id is not null clause"

Package Version

packages:
  - package: fivetran/xero_source
    version: [">=0.3.0","<0.4.0"]

Warehouse

  • [ X] BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Please indicate the level of urgency

Low

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
  • No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Feature] README Updates

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

This packages README is not currently inline with our existing standards. It would be ideal for this repo to see README updates to be in line with our current documentation guidelines.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

During these updates, we should also add the identifier variables and ensure the integration_tests/dbt_project.yml references the identifiers as well.

BUG - [your bug title here]

Are you a current Fivetran customer?
yes

Describe the bug
In the report generated by Xero we have some field values positive for PL but in the report generated by fivetran dbt package
of column net_amount we have those value as negative for some accounts although we have positive values in the source table but the transformation written in the dbt doesn't fetch those values .

Steps to reproduce
1.Go to the Fivetran dashboard and create a connector using Xero as the source and BigQuery as the destination.
2. add transformation for connector using Fivetran Xero DBT package.
3. after running sync you will get reports generated through the DBT package in BigQuery.
4.Now Just compare reports value with original Xero reports.

Expected behavior
we should get the values for net_income positive in the final report
Project variables configuration

name: 'client_name'
profile: 'my-bigquery-db'
version: '0.3.0'
config-version: 2
require-dbt-version: [">=0.20.0"]

vars:
  xero_schema: xero_five_tran
  xero_database: 
  xero_five_tran:
    account: "{{ ref('stg_xero__account') }}" 
    contact: "{{ ref('stg_xero__contact') }}" 
    invoice_line_item: "{{ ref('stg_xero__invoice_line_item') }}" 
    invoice: "{{ ref('stg_xero__invoice') }}" 
    journal_line: "{{ ref('stg_xero__journal_line') }}" 
    journal: "{{ ref('stg_xero__journal') }}" 
    organization: "{{ ref('stg_xero__organization') }}" 
    credit_note: "{{ ref('stg_xero__credit_note') }}" 
    bank_transaction: "{{ ref('stg_xero__bank_transaction') }}" 
  
models:
 :
    +materialized: table
    +schema: xero
    

Package Version
package: fivetran/xero_source
version: [">=0.2.0","<0.3.0"]

packages:
  - package: fivetran/xero_source
    version: [">=0.3.0","<0.4.0"]

Warehouse

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Additional context

Screenshots

Please indicate the level of urgency
This is really important because, the client needs the report, but as I told the report we are generating through the Fivetran DBT package has the wrong net_amount which is blocking us. This will really affect our relationship with the customer. If we don't deliver it on time it may result in fine also.

Are you interested in contributing to this package?
no

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
  • No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Bug] Using BigQuery adapter, validation_errors - Invalid cast from STRING to JSON

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

First, thanks for the great work on these models!

When using BigQuery as the data warehouse for my data and running the XERO transform DBT models, there is one temp step that fails, causing the invoice line items model to be skipped.

The step is stg_xero__invoice_line_item_tmp and the error is in the following line:
cast('validation_errors' as JSON) as 'validation_errors'
(this ends up on line 26 of the rendered SQL in BQ, but for my use case, I am using multiple schemas to create a single set of output models)

The error being returned by BigQuery is the following:
Invalid cast from STRING to JSON at [111:26]
image

(again, the line numbers are likely specific to my project as I have multiple input schemas)


I noticed recently for the XERO connectors (and a few others) in Fivetran whilst using BigQuery as the destination that there was the following notification and wondered if the two were related?
image

I am unsure how to troubleshoot further and support where I can -

Relevant error log or model output

No models are output.

Errors as above in the description.

Final log status from the `dbt run` command as follows:

06:52:23  Completed with 1 error and 0 warnings:
06:52:23  
06:52:23  Database Error in model stg_xero__invoice_line_item_tmp (models/tmp/stg_xero__invoice_line_item_tmp.sql)
06:52:23    Invalid cast from STRING to JSON at [111:26]
06:52:23    compiled Code at target/run/xero_source/models/tmp/stg_xero__invoice_line_item_tmp.sql


### Expected behavior

SQL is compiled appropriately and the [xero__invoice_line_items](https://github.com/fivetran/dbt_xero/blob/main/models/xero__invoice_line_items.sql) table is produced.

### dbt Project configurations

The following are the variables (obfuscated input schemas) in the `dbt_project.yml`:

```yaml
vars:
  xero_source:
    union_schemas: ['xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx','xero_xxxxxxx']

Using threads: 4 in the profile.

No other project specific vars or configurations are used.

Package versions

packages:
  - package: fivetran/xero
    version: [">=0.6.0", "<0.7.0"]

What database are you using dbt with?

bigquery

dbt Version

Core:
  - installed: 1.6.1
  - latest:    1.7.2 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 1.6.3 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

After attempting an update, I get the same issues with the following output from dbt --version

Core:
  - installed: 1.7.2
  - latest:    1.7.2 - Up to date!

Plugins:
  - bigquery: 1.7.2 - Up to date!

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Bug] ERRORS ON TRANSFORMATIONS

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

This pertains to Xero Balance Sheet transformations.

Notice how you use the snapshot financial_year_end_month and financial_year_end_day as the financial year end, so it goes and creates the 29th of February for 2023, 2022, 2021 etc.

Those dates don't exist and create an error:

select 
    case
        when cast(extract(year from current_date) || '-' || financial_year_end_month || '-' || financial_year_end_day as date) >= current_date
        then cast(extract(year from current_date) || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
        else cast(extract(year from {{ dbt.dateadd('year', -1, 'current_date') }}) || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
    end as current_year_end_date,

I've attached the file showing financial_year_end_day and financial_year_end_month as fixed on 02/29.

Relevant error log or model output

23:04:36
Database Error in model xero__balance_sheet_report (models/xero__balance_sheet_report.sql)
100040 (22007): Date '2023-2-29' is not recognized
compiled Code at target/run/xero/models/xero__balance_sheet_report.sql
23:04:36
23:04:36
Done. PASS=17 WARN=0 ERROR=1 SKIP=0 TOTAL=18

Expected behavior

Not to give an error

dbt Project configurations

I don't know how to do this, I'm not a programmer

Package versions

I don't know how to do this, I'm not a programmer

What database are you using dbt with?

snowflake

dbt Version

I don't know how to do this, I'm not a programmer

Additional Context

I'm being sent here by fivetran support, as a business user who does ACCOUNTING, instead of FIvetran handling it. My whole system is offline because of the errors and Fivetran can't give two shits about that.

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

BUG - Reports generated from the Zero DBT package has the wrong net_amount

Are you a current Fivetran customer?
Yes. Farhat Khan, Onplan Co

Describe the bug
We have a Xero connector for our client in Fivetran, our requirement is to fetch reports from Xero, but Fivetran is only receiving schemas, not reports, so for generating reports like Balance Sheet, Profit and Loss, Trial Balance, etc. For this, we are using the Fivetran Zero DBT package. But the report generated through the Fivetran DBT package has the wrong net_amount, all net_amount in the report are wrong.

Steps to reproduce

  1. Go to the Fivetran dashboard and create a connector using Xero as the source and BigQuery as the destination.
  2. add transformation for connector using Fivetran Xero DBT package.
  3. after running synch you will get reports generated through the DBT package in BigQuery.
  4. Now Just compare reports value with original Xero reports.

Expected behavior
Reports generated through Fivetran should have the same value as original Xero reports.

Project variables configuration

name: 'five_tran_xero'
version: '0.2.0'
config-version: 2
require-dbt-version: [">=0.20.0"]

vars:
xero_schema: five_tran_xero
xero_database: etl_data_base
five_tran_xero:
account: "{{ ref('stg_xero__account') }}"
contact: "{{ ref('stg_xero__contact') }}"
invoice_line_item: "{{ ref('stg_xero__invoice_line_item') }}"
invoice: "{{ ref('stg_xero__invoice') }}"
journal_line: "{{ ref('stg_xero__journal_line') }}"
journal: "{{ ref('stg_xero__journal') }}"
organization: "{{ ref('stg_xero__organization') }}"
credit_note: "{{ ref('stg_xero__credit_note') }}"
bank_transaction: "{{ ref('stg_xero__bank_transaction') }}"

models:
five_tran_xero:
+materialized: table
+schema: xero

Package Version

packages:

  • package: fivetran/xero_source
    version: [">=0.2.0","<0.3.0"]

Warehouse

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Additional context

Screenshots
can't provide due to the data security

Please indicate the level of urgency
This is really important because, the client needs the report, but as I told the report we are generating through the Fivetran DBT package has the wrong net_amount which is blocking us. This will really affect our relationship with the customer. If we don't deliver it on time it may result in fine also.

Are you interested in contributing to this package?
No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Feature] Support consolidated currency conversion

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Support transforms reporting their values in the converted currency of the primary subsidiary. Currently they are reported in the source currency of each currency, resulting in mixed values in the output.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

FEATURE - [To be able to generate Employee Remuneration Report present in Xero]

Are you a Fivetran customer?

Yes. Aamir Sheikh, Onplan Co

Is your feature request related to a problem? Please describe.

Yes. We have a feature in our application in which we need to pull this Employee Remuneration Report data from Xero on our platform and allow the user to access it.

Describe the solution you'd like

We created a default Xero connector which is pulling complete snapshot into our bigquery database. From that database we are trying to prepare this report by doing the analysis on the extracted database snapshot.
So I wanted to take help from your support if they can assist us with a DPT package using which we can generate that Employee Remuneration Report.

Describe alternatives you've considered

Create a connector-cloud function and pull that data from API's. But the challenge is, its Oauth 2 based integration and we will have to keep the access token rotated to avoid expiration of it.
So is it possible we can create an oauth connector for Xero which can manage/rotate the oauth credentials and pass on the access tokens to a cloud function.
In that cloud function we will implement the logic to pull data through XERO API's using the access tokens passed by Fivetran connector.

Additional context

Please indicate the level of urgency and business impact of this request

**It is really critical and it is one of the crucial data we are supposed to pull from xero portal. And show to our customer in our platform. It would really affect our relationship with the customer. It might result penalty as well if we will not deliver it in time. **

Are you interested in contributing to this package?

No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

[Bug] Bank Transaction table missing

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

One of our 3 Xero instances doesn't have the bank_transaction table and most models are skipped because of this.

Relevant error log or model output

No response

Expected behavior

If the table doesn't exist, users should be able to not include it in the models being ran through a variable, just like the credit_note

dbt Project configurations

We just use the standard configuration for this package.

models:
xero:
+tags: 'default_transformation_xero'
+schema: "dataset"
xero_source:
+tags: 'default_transformation_xero'
+schema: "dataset"

Package versions

  • package: fivetran/xero
    version: 0.4.0

What database are you using dbt with?

bigquery

dbt Version

version: 1.0.1

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Bug] Balance Sheet duplicating totals when multiple companies are consolidated

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

When using this package to consolidate multiple companies, the account values are erroneously multiplied by the number of companies in the balance sheet report.

For example, if three companies are combined in this package, the net_amount value in the xero__balance_sheet_report model is tripled.

Relevant error log or model output

Consolidate multiple companies using union_schemas in dbt_project.yml

Then run 
select * from {{ ref('xero__balance_sheet_report') }}

Then compare to Xero or source data and see that the net_amounts are incorrect.

Expected behavior

The net_amounts should equal the balance sheet report in Xero.

Instead of this, they are multiples higher.

dbt Project configurations

xero_source:
union_schemas: ['company1_xero','company2_xero','company3_xero']

Package versions

packages:

  • package: fivetran/xero
    version: [">=0.4.0", "<0.5.0"]
  • package: dbt-labs/dbt_utils
    version: 0.8.4

What database are you using dbt with?

bigquery

dbt Version

1.1

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

FEATURE - [Add Purchase Orders to Xero packages]

Are you a Fivetran customer?

Yes, Doug Jeffery, Principal Data Architect, Modulous

Is your feature request related to a problem? Please describe.

The Modulous FD has asked me if we can report on Purchase Orders in Xero - I can see they are synched from Xero into Snowflake but they are not considered in the dbt transformations as far as I can tell

Describe the solution you'd like

Extend the fivetran dbt xero and dbt xero source packages to create a transformed view of Purchase Orders

Describe alternatives you've considered

Extending the packages myself

Additional context

Please indicate the level of urgency and business impact of this request

It would keep my FD happy but not super urgent tbh

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
  • No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

[Bug] Test Failure in dbt xero package

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

I am getting an error on this test:
dbt_utils_unique_combination_of_columns_xero__balance_sheet_report_source_relation__date_month__account_name (models/xero.yml)

Adding account_id to the test seems to fix it.

Relevant error log or model output

Failure in test dbt_utils_unique_combination_of_columns_xero__balance_sheet_report_source_relation__date_month__account_name (models/xero.yml)

Expected behavior

The test should pass for the test Failure in test dbt_utils_unique_combination_of_columns
on table xero__balance_sheet_report (dbt_utils_unique_combination_of_columns_xero__balance_sheet_report_source_relation__date_month__account_name (models/xero.yml))

dbt Project configurations

N/A

Package versions

  • package: fivetran/xero
    version: [">=0.4.0", "<0.5.0"]

What database are you using dbt with?

snowflake

dbt Version

1.1.2

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Bug] Unique combination test failing - impacts production deployment

Is there an existing issue for this?

  • I have searched the existing issues

Describe the issue

I am seeing this error when I run my production deployment job:
FAIL 1 dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation

which means that these models are skipped:

xero__balance_sheet_report .......... [SKIP]
xero__profit_and_loss_report ........ [SKIP]

and these tests are skipped:

dbt_utils_unique_combination_of_columns_xero__balance_sheet_report_source_relation__date_month__account_name
not_null_xero__profit_and_loss_report_profit_and_loss_id
unique_xero__profit_and_loss_report_profit_and_loss_id

The error is:
23:09:33 Completed with 1 error and 0 warnings: 23:09:33 23:09:33 23:09:33 Failure in test dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation (models/xero.yml)

and when I download the compiled code I get

`with validation_errors as (

select
    journal_line_id, source_relation
from insights.matchandwood_xero.xero__general_ledger
group by journal_line_id, source_relation
having count(*) > 1

)

select *
from validation_errors
`

if I run that directly in Snowflake I get 1 record returned:

JOURNAL_LINE_ID is null, SOURCE_RELATION is an empty string (I think, it doesn't say NULL) and a count of 4 records.

Relevant error log or model output

Failure in test dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation (models/xero.yml)

Expected behavior

Job to complete successfully, no models skipped

dbt Project configurations

Values set for xero_schema and xero_database

Package versions

packages:

packages:

  • package: fivetran/xero
    version: [">=0.6.0", "<0.7.0"]

What database are you using dbt with?

snowflake

dbt Version

v.1.6

Additional Context

No response

Are you willing to open a PR to help address this issue?

  • Yes.
  • Yes, but I will need assistance and will schedule time during our office hours for guidance
  • No.

[Feature] Xero Cash Summary report in DBT

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Hi,

I am looking for help, guidance, or documentation on how to implement this report: https://central.xero.com/s/article/Cash-Summary into dbt_xero so I can have a table with the report generated in dbt.

Describe alternatives you've considered

I haven't really considered any alternatives. My first idea was to see if this could be done with the existing package.

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

I am not sure about contributing this feature, because I don't know how complicated it is to do this. But I am open to discussing this further.

I am actually looking for a possibility to add some more reports from xero to dbt, so this could be a good first report I guess.

BUG - All Xero Dbt jobs failing due to missing table

Are you a current Fivetran customer?
Yes. Shubham Sinha, Head of Engineering, Daydream

Describe the bug
dbt run fails since, table credit_note doesn't exist for three separate accounts. There are no config vars that I could use to add a true/false flag to skip this table.

Attaching Bug report from dbt run:

Database Error in model stg_xero__credit_note_tmp (models/tmp/stg_xero__credit_note_tmp.sql)
  relation "trainwithkickoff_39_xero.credit_note" does not exist
  LINE 4:     select * from "cloud_services"."trainwithkickoff_39_xero...
                            ^
  compiled SQL at target/run/xero_source/models/tmp/stg_xero__credit_note_tmp.sql

Steps to reproduce
Hard to reproduce this locally unless you have the same setup (account, tables) but I believe the bug description should have all the details.

Expected behavior
Possible to create some config vars that can be used to skip this table?

Project variables configuration

vars:
  xero_schema: trainwithkickoff_39_xero
  xero_database: cloud_services

models:
  xero:
    +schema: transformations

Package Version

packages:
  - package: fivetran/xero
    version: 0.1.1

Warehouse

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Additional context

Screenshots
Attaching all the tables synced from Account 1 (Real production users):
Screen Shot 2021-06-22 at 1 10 58 AM

Attaching all the tables synced from Account 2 (Real production users):
Screen Shot 2021-06-22 at 1 19 38 AM

Please indicate the level of urgency
It's pretty critical for us as it affects several Production users.

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
  • No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

BUG - unique_xero__profit_and_loss_report_profit_and_loss_id fails when account_name column differs

Are you a current Fivetran customer?

Yes - Jared - Samarkand Global - Data Team Lead

Describe the bug

There is a test on the profit and loss report -

  - name: profit_and_loss_id
    description: Unique identifier for each record in the profit and loss report model.
    tests:
      - unique  
      - not_null

That tests for a unique profit_and_loss_id

However, this test fails when there is a difference in the account_name column.

This difference occurs when the account_name is changed in the Xero back office system.

This change does not update historic entries therefore you may end up with something like the following:

Advertising & Marketing
Advertising

When the profit_and_loss_id is generated it does not use the account_name. Therefore you end up with a duplicate profit_and_loss_id

Expected behavior

profit_and_loss_id to be unique in rows in the profit_and_loss_model

Package Version

packages:
  - package: fivetran/xero_source
    version: [">=0.3.0","<0.4.0"]

Warehouse

  • [ X] BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Screenshots
image
Please indicate the level of urgency

Low. Reports not customer facing yet but trying to ensure the package works as expected before creating dashboards.

Are you interested in contributing to this package?

  • Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
  • No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation fails when journal line id is NULL

There are instances where there are rows in the xero__general_ledger model that have a NULL value in the journal_line_id column.

This causes the test:

dbt_utils_unique_combination_of_columns_xero__general_ledger_journal_line_id__source_relation

because the NULL values are seen to be the same journal_line_id

Suggestion

We could add journal_id or journal_number to the relations test. However, there still might be a case where there is a journal with more than one row in the model that has a NULL value in journal_line_id

[Feature] Databricks compatability

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Internal - Add databricks compatibility

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

  • Yes.
  • Yes, but I will need assistance and will schedule time during your office hours for guidance.
  • No.

Anything else?

No response

Feature Request -- Join Contact details to General Ledger

Hi Kristin -

After getting in and testing out the data I just had one quick question.

Is there any way we would be able to add in one detail to the General Ledger information? We currently have line description, journal reference, and account name but it would be helpful if we could pull in the contact related to the General Ledger line. Do you know if this would be possible at all? I was hoping to be able to use it as a filter. The only info we can filter by contact is the invoice detail currently so it would be helpful to have this addition.

Thanks!

FEATURE - [Allow to Combine Multiple Xero Companies]

Are you a Fivetran customer?
Fernando Tomas Conticello, Data Consultant, Ocula

Is your feature request related to a problem? Please describe.
I want to have a feature to allow run the packages (Xero and Xero_Source) for multiple companies using the fivetran xero connector.
At the moment the package only allows to run one company.
Describe the solution you'd like
Solution is similar to https://blog.montrealanalytics.com/how-to-use-the-shopify-dbt-package-by-fivetran-with-multiple-shops-eb4cea326ee0

Describe alternatives you've considered
-Generate union base of companies
-Allow Xero_Sources to have pass through column variables
-Allow Xero to have pass through column variables
-Modify the queries of xero models to allow grouping by those pass through column as well. There is a group by (7) in some Xero models that need to be modified to allow other columns coming from the pass through columns

Additional context

Please indicate the level of urgency and business impact of this request
Important

Are you interested in contributing to this package?

  • [X ] Yes, I can do this and open a PR for your review.
  • Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
  • No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.