slidoapp / dbt-superset-lineage Goto Github PK

Because dbt-superset-lineage (0.2.1) depends on sqlfluff (>=0.8.2,<0.9.0)
and no versions of dbt-superset-lineage match >0.2.1,<0.3.0, dbt-superset-lineage (>=0.2.1,<0.3.0) requires sqlfluff (>=0.8.2,<0.9.0).
So, because seed depends on both dbt-superset-lineage (^0.2.1) and sqlfluff (^1.4.1), version solving failed.

Hi! I want to start a conversation about having this library also support a feature to create a dataset if one does not exist in Superset, perhaps as part of the push_descriptions command or an entirely new command, say, create_datasets.

Automatically creating datasets would solve for two use-cases:

1. Ensuring any new dbt models are synced into Superset without having to explicitly create it in Superset itself.
2. Helps sync existing dbt models into a freshly provisioned Superset instance -- again reduces effort to create the corresponding datasets in Superset (via a separate script or manual actions in the Superset UI)

This PR (rohitsanj#1) against my own fork of this repo introduces a new flag create_dataset_if_not_exists to the push_descriptions command. I've also added in a new folder called dbt_schemas containing the dbt manifest JSON schema and the schema-generated pydantic models -- this is used to parse the dbt manifest.json file to provide helpful type hints when developing and automatic data validation at runtime.

Would love to know the community's thoughts on this and if others have come across the requirement for such a feature. Thanks!

Hi,

thank you for the great package. would something like this also work with Metabase?

Thank you

Hi team. First off, thanks for the great project.

We generated a token via the API (/security/login) which includes an access + refresh token. We stored the output of these as secure credentials in our CI environment. Things have been working great with this script in CI for the past 2 weeks. Then, all of a sudden we're getting 401 Unauthorized responses for /security/refresh calls.

Nothing has changed in our environment / users-wise.

I'm curious if the refresh tokens only last for 2 weeks now? Is there a way to generate longer lived API tokens somehow?

This issue is likely outside the scope of this project per-say; however, I couldn't find very detailed information on the API access tokens anywhere else and was curious if others had experienced this or had a workaround to generate tokens that could persist in CI at some longer/indefinite period.

/ping @one-data-cookie

/cc @bkimjin

In an effort to ease user flow from dbt docs into Superset, we could programmatically append a markdown hyperlink into a dbt model's description, pointing to the "create chart" page in Superset.

For example, consider the following file, orders.yml

# orders.yml
models:
  - name: orders
    description: |
      This table has basic information about orders, as well as some derived facts based on payments

      [Explore this table!](http://localhost:8088/explore/?datasource_type=table&datasource_id=97)
    columns: ...

dbt docs page:

Clicking "Explore this table!" takes you to this Superset page:

#32 states that SQLFluff should have been updated to 2.1.2, but it seems like dbt-superset-lineage is still referencing 1.4.5.

Commands On a Clean Virtual Instance

pip list

Package    Version
---------- -------
pip        22.3.1
setuptools 68.0.0
wheel      0.38.4

pipenv install dbt-superset-lineage

Package              Version
-------------------- ---------
...
setuptools           68.0.0
soupsieve            2.4.1
sqlfluff             1.4.5
tblib                2.0.0
toml                 0.10.2
...

Hey guys,
I adapted the pull_dashboards.py file to use it as, when trying to run the code out of the box, I was not getting the dashboards' table names.

Out of my interpretation GET /dashboard/id doesn't return "table_name" anymore and it was replaced by a similar command I found (GET /dashboard/id/datasets).

Let me know how this sounds to you, I am happy to contribute with my minor edits.
Best,
Agus

Hi, is this repo still actively maintained. I have been testing this on a Docker deployment of Superset and unable to disable CSRF tokens. there are a number of threads on this which seem to indicate this is a common issue. Is there a way to ensure this library works for POST commands with SS or any specific steps to disable CSRF.

I have noticed that only column descriptions are pushed to superset, I think it would be great if also Labels were included.