grai-io / grai-core Goto Github PK

License: Other

Dockerfile 0.09% Python 69.71% Makefile 0.13% Shell 0.27% HTML 5.97% CSS 0.01% TypeScript 23.77% TSQL 0.01% JavaScript 0.03%

hacktoberfest data datalineage dbt open-source postgresql python mssql mysql snowflake

grai-core's Introduction

$Python fraction of codebase$

Documentation • Website • Slack • Discussion • Want to Chat?

Introduction

Data lineage made simple. Grai makes it easy to understand and test how your data relates across databases, warehouses, APIs, and dashboards.

Pre-built connectors. Automatically synchronize lineage from across the stack so your metadata is never out of date.
Centralized data tests. Write data validation tests that run whenever upstream data sources change.
Integrated with GitHub. Run data validation tasks as part of your CI/CD process to test changes everywhere your data is used.
Your data, your cloud. is fully open source and self-hosted. You maintain full control over your data and hosting environment.

How it works

Automatically build column level lineage spanning your warehouse and production services with connectors for dbt, Snowflake, Fivetran, and more (see below).
Get alerts in your CI/CD workflows whenever changes to a production system will impact your warehouse or dbt projects with GitHub Actions.
Self-host the project or run it in the Grai Cloud for free.

Connectors

	integration	install
	Snowflake	`pip install grai-source-snowflake`
	BigQuery	`pip install grai-source-bigquery`
	Redshift	`pip install grai-source-redshift`
	Postgres	`pip install grai-source-postgres`
	MySQL	`pip install grai-source-mysql`
	SQL Server	`pip install grai-source-mssql`
	dbt	`pip install grai-source-dbt`
	Fivetran	`pip install grai-source-fivetran`
	csv, parquet, feather	`pip install grai-source-flat-file`
	Metabase	`pip install grai-source-metabase`
	Looker (alpha)	`pip install grai-source-looker`

Quickstart

You can find a full quickstart guide in the documentation which covers deploying your own instance of Grai and getting set up with your first connector in Python. The fastest way to get started is through the Grai CLI but you can also run the project locally with docker compose.

Default login credentials:

username: [email protected]
password: super_secret

CLI

pip install grai-cli
grai demo start

Running Locally

You can always find pre-built images of the backend server at ghcr.io/grai-io/grai-core/grai-server:latest and the frontend at ghcr.io/grai-io/grai-core/grai-frontend:latest, however, if you prefer to build from source, you can do so with docker compose.

git clone https://github.com/grai-io/grai-core
cp examples/deployment/docker-compose/docker-compose.yml
docker compose up

The backend server will be available at http://localhost:8000/ and the frontend is now here http://localhost:3000/.

After logging in and connecting a data source you'll be greeted with a lineage graph looking something like this

For more information about using the web application check out the getting started guide.

Other Deployment Mechanisms

You can find example configurations for docker compose and Kubernetes in the examples folder.

Helm

We also publish a set of Helm charts which are available if you prefer.

helm repo add grai https://charts.grai.io
helm install grai grai/grai

Component Services

grai-server: The backend metadata service built on Postgres and Django as the Metadata persistence layer.
grai-frontend: The frontend web application built on React.
grai-cli: Python client library for interacting with the Grai server.
grai-schemas: The python metadata schema implementation library of Grai. It provides a standardized view of all Grai objects used to ensure compatibility between the server, integrations, and the client.
grai-graph: A python utility library for working with the Grai metadata graph.
grai-actions: A library of GitHub Actions implementations to integrate Grai tests into your CI/CD pipelines.
integrations: A collection of integration libraries to extract metadata and persist their results to Grai.

Community Roadmap

Community Feedback drives our roadmap. Please let us know what you'd like to see next by asking questions and upvoting feature requests!

Repo Activity

Community

Email us: [email protected]

Join us on Slack:

Check us out at www.grai.io

grai-core's People

Contributors

Stargazers

Watchers

grai-core's Issues

Add a default ordering on Django ViewSets

The current Django ViewSets do not include ordering or ordering_fields settings which can affect paginated endpoints.

These should be set on

When complete, tests will no longer generate UnorderedObjectListWarning's e.g.

workspaces/tests/test_views.py::TestMemberships::test_get_memberships
workspaces/tests/test_views.py::TestMemberships::test_get_memberships_filter_by_is_active
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/rest_framework/pagination.py:200: UnorderedObjectListWarning: Pagination may yield inconsistent results with an unordered object_list: <class 'workspaces.models.Membership'> QuerySet.
    paginator = self.django_paginator_class(queryset, page_size)

[GRA-909] Add html to email for verification

Current email template is at https://github.com/grai-io/grai-core/blob/af7a53ff85e0cd8b47698aca55f44897d9f583b6/grai-server/app/templates/auth/email_verification.txt but is text only, would be nice to add an html template.

I presume that Django has an off the shelf way to do this.

The current email could be improved with more text, sign off and so on.

_GRA-909

Failed transactions in the client don't roll back

If the client is tasked with updating a batch of resource but fails midway through the previously executed transactions remain unchanged.

[GRA-937] Add html to password verification email

Update password verification email to send both plain text and html content.

_GRA-937

[GRA-1072] [GRA-1073] Example docker compose files out of date

null

_GRA-1073

Postgres tests fail on empty database

Tests currently rely upon the assumption of a populated database but it should be possible to run tests against an empty database.

[GRA-258] Pressing run on an existing run with a RunFile fails

To reproduce:

Add a source which uses a file, for example dbt or yaml file.
You should get a screen showing the run, as the file is processed.
On this screen press the Run button and you will get an error.

The bug relates to the fact that the file is associated with an individual run, as a RunFile Django model. Pressing the button causes a new Run to be created, but it isn't associated with the file. The solution is to copy the file and create a new RunFile for the new Run.

_{From SyncLinear.com | GRA-258}

Integrate pull request preview environments

I would like to support Grai by implementing Uffizzi preview environments.
Disclaimer: I work on Uffizzi.

Uffizzi is a Open Source full stack previews engine and our platform is available completely free for Grai (and all open source projects). This will provide maintainers with preview environments of every PR in the cloud, which enables faster iterations and reduces time to merge. You can see the open source repos which are currently using Uffizzi over here

Uffizzi is purpose-built for the task of previewing PRs and it integrates with your workflow to deploy preview environments in the background without any manual steps for maintainers or contributors.

We can go ahead and create an Initial PoC for you right away if you think there is value in this proposal.

test

[GRA-908] [GRA-548] Improved test failures email

Currently just says "Test Failures"

_{From SyncLinear.com | GRA-548}

_GRA-908

`'NoneType' object has no attribute 'encode'` on homepage search bar

Hi,

Running Grai locally and after creating a few nodes/edges, I still see this error as soon as I click the search bar on the homepage.

Here are the logs:

grai-server-frontend-1            | 172.19.0.1 - - [04/May/2023:21:16:52 +0000] "GET /default/default HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
grai-server-frontend-1            | 172.19.0.1 - - [04/May/2023:21:16:52 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "http://localhost:3000/default/default" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
grai-server-the_guide-1           | 'NoneType' object has no attribute 'encode'
grai-server-the_guide-1           |
grai-server-the_guide-1           | GraphQL request:4:5
grai-server-the_guide-1           | 3 |     id
grai-server-the_guide-1           | 4 |     search_key
grai-server-the_guide-1           |   |     ^
grai-server-the_guide-1           | 5 |     __typename
grai-server-the_guide-1           | Traceback (most recent call last):
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/graphql/execution/execute.py", line 528, in await_result
grai-server-the_guide-1           |     return_type, field_nodes, info, path, await result
grai-server-the_guide-1           |                                           ^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry_django_plus/utils/aio.py", line 89, in resolve_async
grai-server-the_guide-1           |     ret = resolver(await value)
grai-server-the_guide-1           |                    ^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__
grai-server-the_guide-1           |     ret = await asyncio.wait_for(future, timeout=None)
grai-server-the_guide-1           |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
grai-server-the_guide-1           |     return await fut
grai-server-the_guide-1           |            ^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 22, in run
grai-server-the_guide-1           |     result = self.fn(*self.args, **self.kwargs)
grai-server-the_guide-1           |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 490, in thread_handler
grai-server-the_guide-1           |     return func(*args, **kwargs)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/types/fields/resolver.py", line 187, in __call__
grai-server-the_guide-1           |     return self.wrapped_func(*args, **kwargs)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/src/app/api/types.py", line 580, in search_key
grai-server-the_guide-1           |     return client.generate_secured_api_key(
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/src/app/api/search.py", line 6, in generate_secured_api_key
grai-server-the_guide-1           |     return SearchClient.generate_secured_api_key(parent_api_key, restrictions)
grai-server-the_guide-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/algoliasearch/search_client.py", line 275, in generate_secured_api_key
grai-server-the_guide-1           |     parent_api_key.encode("utf-8"),
grai-server-the_guide-1           |     ^^^^^^^^^^^^^^^^^^^^^
grai-server-the_guide-1           | AttributeError: 'NoneType' object has no attribute 'encode'
grai-server-the_guide-1           | Stack (most recent call last):
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
grai-server-the_guide-1           |     self._bootstrap_inner()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
grai-server-the_guide-1           |     self.run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/threading.py", line 975, in run
grai-server-the_guide-1           |     self._target(*self._args, **self._kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
grai-server-the_guide-1           |     work_item.run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
grai-server-the_guide-1           |     result = self.fn(*self.args, **self.kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 248, in _run_event_loop
grai-server-the_guide-1           |     loop.run_until_complete(coro)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
grai-server-the_guide-1           |     self.run_forever()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
grai-server-the_guide-1           |     self._run_once()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
grai-server-the_guide-1           |     handle._run()
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
grai-server-the_guide-1           |     self._context.run(self._callback, *self._args)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 306, in main_wrap
grai-server-the_guide-1           |     result = await self.awaitable(*args, **kwargs)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/django/views.py", line 254, in dispatch
grai-server-the_guide-1           |     return await self.run(request=request)
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/http/async_base_view.py", line 176, in run
grai-server-the_guide-1           |     result = await self.execute_operation(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/http/async_base_view.py", line 115, in execute_operation
grai-server-the_guide-1           |     return await self.schema.execute(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/schema/schema.py", line 248, in execute
grai-server-the_guide-1           |     result = await execute(
grai-server-the_guide-1           |   File "/usr/local/lib/python3.11/site-packages/strawberry/schema/execute.py", line 156, in execute
grai-server-the_guide-1           |     process_errors(result.errors, execution_context)

CLI command produces recursion error

The command grai config view currently generates a RecursionError in the config's __getattr__.

Tests using timezone naive datetime for timezone aware field

The following warning messages are being emitted for timezone naive test values:

connections/tests/test_tasks.py::TestEventsAllTests::test_dbt_cloud
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1535: RuntimeWarning: DateTimeField Event.date received a naive datetime (2023-10-09 00:00:00) while time zone support is active.
    warnings.warn(

auth/tests/test_mutations.py::test_verify_email
  /Users/ian/Library/Caches/pypoetry/virtualenvs/the-guide-y4fToIIm-py3.11/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1595: RuntimeWarning: DateTimeField User.verified_at received a naive datetime (2023-10-09 14:25:20.943758) while time zone support is active.
    warnings.warn(

These tests should be modified so that running tests no longer emits the respective warning message.

Change `NGROK_URL` environment variable

This should be server_url or similar

ClickHouse support?

Thanks for the great tool, would you like to add ClickHouse support?

https://clickhouse.com/docs/en/

it have lot of connectors
https://github.com/xzkostyan/clickhouse-sqlalchemy.git
https://github.com/mymarilyn/clickhouse-driver

https://github.com/ClickHouse/clickhouse-connect

Create documentation for demo environment configs

Various docker deployment configs can be found in the examples folder alongside the cli command grai demo start. However these are largely undocumented. We should include more details in the various doc pages including

a copy of the docker scripts
A page about the cli demo commands

twisted.web has disordered HTTP pipeline response

This issue refers to Dependabot alert #155 https://github.com/grai-io/grai-core/security/dependabot/155

Redshift integration not detecting late-binding view columns

Description

Running the Redshift integration against databases with late-binding views fails to generate columns associated with late-binding views. Although referenced tables appear in information_schema.tables their associated columns do not appear in information_schema.columns resulting in an error here.

Proposed Solution

It appears possible to collect metadata from late binding views via pg_get_late_binding_view_cols which should be included alongside information_schema.columns.

Originally reported on slack

Django USE_L10N setting is deprecated

The django USE_L10N setting is being deprecated and needs to be removed from settings.

[GRA-454] Organisation names with slash break routing

There is nothing to stop a user creating an organisation with one or more slashes in it. These break the Web App routing, which relies on slashes.

The same issue would occur with workspace names.

_GRA-454

gunicorn worker timeout

Bug report was issued by Vignesh Hari. He's seeing worker timeout issues usually on boot but sporadically afterwards as well. He's running on iOS under apple silicon.

[GRA-464] Lineage to multiple foreign keys is omitted

We are currently bypassing foreign keys in the RDBMS integrations which key to multiple columns like can be found here: https://dba.stackexchange.com/questions/240160/postgresql-multiple-column-foreign-key

We should update integrations to handle this scenario.

_GRA-464

GraphQL APIs need Documentation

Although the REST API's have both swagger and redoc pages, the GraphQL API's are undocumented. It would be nice to include something like graphql playground but other options are possible.

The set of graphql implementations are available here