turntable-so / turntable Goto Github PK

Open-Source Data Platform

Metrics, Models, Jobs, Notebooks, End-to-end Column Lineage, and More!
Define your source of truth with an open standard
Slack · Website · Issues · Roadmap

The problem: Today's data stack is highly fragmented, insecure, and expensive.

• Most modern data teams have at least 5-10 tools in their data stack. They often don't work well together.
• Today's closed ecosystem is vulnerable to security breaches and tool discontinuation.
• Vendor lock-in has led to exorbitant costs and rent-seeking behavior:

The Solution: Turntable, a post-modern data stack

• Open: Our whole repo is AGPL and always will be.
• Private: Self-deploy our open-source version or use our cloud product with our customer-deployed agent. Either way, your data never leaves your infrastructure.
• Integrated: With Turntable, all of your metrics, models, and data documents exist in the same context. We think this will allow for some totally new experiences. Think clicking on a dashboard and seeing exactly where a number came from.

• End-to-end Column lineage: See your data's journey from source to dashboard.
• Data catalog: See all your data assets in one place. We event write AI-generated descriptions for you (opt-in of course).
• Jobs (coming soon): Schedule and monitor your data pipelines.
• [] Pages (coming soon): Write and schedule analyses, reports, and data documents in an AI-powered, Notion-like interface.
• Full observability suite: Detect anomalies, monitor data quality, reduce cost, secure PII and more.
• Metrics: Define once, use anywhere, in pure SQL and GUI. No need to learn a new semantic layer framework.
• Subscriptions: Subscribe to data assets and get notified when they change. Each user gets their own dashboard
• Models: Allow team members to schedule and publish sql queries that depend on one another, with best practices already built in.
• CLI: Use the code editor you already love, but make secure onboarding a breeze.

Data Sources: Postgres, Snowflake, BigQuery, DuckDB, Redshift, S3, and more.

BI Tools: Looker, Tableau, Metabase, PowerBI.

Data Transformation: dbt Core, dbt Cloud, SqlMesh

Saas (coming soon): Stripe, Zendesk, Hubspot, Salesforce, and more.

Turntable launched its v0.1 on August 2024. Lots of new features are coming, and are generally released on a bi-weekly basis. Watch updates of this repository to be notified of future updates.

Check out our public roadmap

Our repo stands on the shoulders of some awesome open source projects. A big reason we decided to go open source is that there are lots of players in the data space who wrap open source projects in a proprietary layer and charge a lot for it. That didn't feel right to us.

With that context, we wanted to explicitly call out some of the projects we love and how we use them:

• Ag-Grid: Table component
• DataHub: metadata ingest engine
• Django: Core application framework
• Hatchet:*workflow scheduler
• Ibis: metrics layer and database connectors
• Instructor: structured AI output
• Lago: inspiration for our documentation
• Mintlify: docs site generator
• Nivo: charting
• Posthog: analytics
• Reactflow: lineage canvas
• Recharts: charting
• Rye: package management
• SQLGlot: sql/lineage parser
• TipTap: Text editor

Turntable is distributed under the AGPLv3 License.

1. Install Docker on your machine;
2. Make sure Docker Compose is installed and available (it should be the case if you have chosen to install Docker via Docker Desktop); and
3. Make sure Git is installed on your machine.

To start using Turntable

1. Clone the repository

# Get the code
git clone https://github.com/turntable-so/turntable.git

# Go to Turntable folder
cd turntable

2. Configure environment variables Create a .env file in the root of the project by running the following command

MacOS or Linux

bash generate_keys.sh

Windows

powershell -executionpolicy bypass -File .\generate_keys.ps1

No environment variables except the secrets generated by the commands above are required to run the app, but some functionality may be limited (e.g. AI-written documentation). See .env.example for a list of all available environment variables.

3. Start the app

To start the app you have two choices:

(A) Run Turntable with No demo resources

Run the following command:

docker compose --env-file .env up

Once the docker build is complete (a few minutes), you will see a line in the terminal like this: 'The app is ready! Visit http://localhost:3000/ to get started'. Once you do, open your browser and go to http://localhost:3000 to see the app running. Signup with a username and password to start using the app.

(B) Run Turntable with demo resources

If you'd like to also see the product with a demo postgres, dbt project, and metabase already connected, run:

docker compose --env-file .env -f docker-compose.demo.yml up --build

Once the docker build is complete (longer than above, usually 5+ minutes), you will see a line in the terminal like this: 'The app is ready! Visit http://localhost:3000/ to get started'. Once you do, open your browser and go to http://localhost:3000 to see the app running. The demo resources can be found in an account with user [email protected] and password mypassword. Login with these credentials to see the demo resources, with associated lineage and asset viewer. If you'd like to start from a blank slate on this instance, simply sign up with a different email.

Please note that Turntable, by default, tracks basic actions performed on your self-hosted instance, but you can easily opt out by setting the value of NEXT_PUBLIC_POSTHOG_KEY to "" in the docker-compose yml file you are using (e.g. docker-compose.yml or docker-compose.demo.yml). We do not track any telemetry in development (i.e. using docker-compose.dev.yml). For more information, please see our privacy policy.

Email us or visit our website to get started with our cloud product. Their our two vairants: a fully-hosted offering, and hybrid one, which includes a customer-deployed agent.

• See the documentation to learn more about all the features;
• Join our Slack community if you need help, or want to chat, we’re here to help;
• Follow us on Twitter or LinkedIn for the latest news;
• You can email us as well: [email protected].

You can follow the instructions below to set up ourm development environment on your machine. This is intended for people interested in contributing to Turntable. If you just want to try Turntable on your local system, we recommend that you the instructions above to run a prod instance.

For now, we are not accepting pull requests from the community, but We are working on a process to make this possible in the future. For now, file an bug reports and feature requests as a GitHub issue, Canny submission or Slack post.

To start the development environment, simply follow the instructions above to start the app, but change the final commmand to:

docker compose -f docker-compose.dev.yml --env-file .env up --build

Unlike the production environemnt, this supports hot reload. It also includes the demo resources described above.

Once everything starts (several minutes), run the command below to access a shell inside the container:

docker compose exec worker bash

From here, backend tests can be run with pytest.