This starter demonstrates using Python alongside a medium-sized dbt project. It uses dbt's jaffle_shop
, dagster-dbt
, and DuckDB.
New to Dagster? Learn what Dagster is in Dagster's hands-on Tutorials or learn using dbt with Dagster in the dbt + Dagster tutorial.
The easiest way to spin up your Dagster project is to use Dagster Cloud Serverless. It provides out-of-the-box CI/CD and native branching that make development and deployment easy.
Check out Dagster Cloud to get started.
To install this example and its Python dependencies, run:
pip install -e ".[dev]"
Then, start the Dagit web server:
dagit
Open http://localhost:3000 with your browser to see the project.
When developing pipelines locally, be sure to click the Reload definition button in the Dagster UI after you change the code. This ensures that Dagster picks up the latest changes you made.
You can reload the code using the Deployment page:
Or from the left nav or on each job page:
Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.
Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.
Check out Using environment variables and secrets for more info and examples.
If you're running Dagster locally and trying to set up schedules, you will see a warning that your daemon isnβt running.
π Expand to learn how to set up a local daemon
If you want to enable Dagster Schedules for your jobs, start the Dagster Daemon process in the same folder as your workspace.yaml
file, but in a different shell or terminal.
The $DAGSTER_HOME
environment variable must be set to a directory for the daemon to work. Note: using directories within /tmp may cause issues. See Dagster Instance default local behavior for more details.
In this case, go to the project root directory and run:
dagster-daemon run
Once your Dagster Daemon is running, the schedules that are turned on will start running.
You can specify new Python dependencies in setup.py
.
Tests are in the assets_dbt_python_tests
directory and you can run tests using pytest
:
pytest assets_dbt_python_tests