A dagster plugin that allows you to run Meltano using Dagster.
You can install using pip install dagster-meltano.
An example of automatically loading all jobs and schedules from your Meltano project.
from dagster import repository
from dagster_meltano import load_jobs_from_meltano_project
meltano_jobs = load_jobs_from_meltano_project("<path-to-meltano-root>")
@repository
def repository():
return [meltano_jobs]An example of running an abitrary meltano run command.
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op
@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
tap_done = meltano_run_op("tap-1 target-1")()
meltano_run_op("tap-2 target-2")(tap_done)
@repository()
def repository():
return [meltano_run_job]You can inject Meltano config with the following Dagster config.
ops:
tap_smoke_test_target_jsonl:
config:
env:
TAP_SMOKE_TEST_STREAMS: '[{"stream_name": "new-stream", "input_filename": "demo.json"}]'An example of running an arbitrary Meltano command.
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_command_op
@job(resource_defs={"meltano": meltano_resource})
def meltano_command_job():
meltano_command_op("install loader tap-smoke-test")()
@repository()
def repository():
return [meltano_command_job]- Open this repository in Visual Studio Code.
- Install the Remote - Containers plugin for Visual Studio Code.
- Go to the example Meltano project root
cd meltano_project - Install all plugins
meltano install - Start dagit
meltano invoke dagster:start - Visit
localhost:3000to access Dagit.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent