prefecthq / prefect-firebolt Goto Github PK

View Code? Open in Web Editor NEW

4.0 6.0 0.0 1.71 MB

Prefect integrations for working with Firebolt data warehouses.

Home Page: https://prefecthq.github.io/prefect-firebolt/

License: Apache License 2.0

Python 100.00%

firebolt prefect

prefect-firebolt's Introduction

prefect-firebolt

Welcome!

prefect-firebolt is a collection of Prefect integrations for working with Firebolt data warehouses.

Firebolt is a Cloud Data Warehousing solution that helps its users streamline their Data Analytics and access to insights. For more information about getting started with Firebolt, read Firebolt's getting started guide.

Getting Started

Python setup

Requires an installation of Python 3.7+.

We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.

These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the Prefect documentation.

Installation

Install prefect-firebolt with pip:

pip install prefect-firebolt

Then, register to view the Firebolt blocks on Prefect Cloud:

prefect block register -m prefect_firebolt

Note, to use the load method on Blocks, you must already have a block document saved through code or saved through the UI.

Write and run a flow

Execute a query against a Firebolt database:

from prefect import flow

from prefect_firebolt import FireboltCredentials, FireboltDatabase, query_firebolt


@flow
def run_firebolt_query():
    firebolt_database_block = FireboltDatabase(
        database="travel",
        credentials=FireboltCredentials(
            username="[email protected]", password="dont42panic"
        ),
    )

    results = query_firebolt(
        database=firebolt_database_block,
        query="SELECT * FROM ex_intergalactic_trips LIMIT 100",
    )

    return results


run_firebolt_query()

Resources

If you encounter any bugs while using prefect-firebolt, feel free to open an issue in the prefect-firebolt repository.

If you have any questions or issues while using prefect-firebolt, you can find help in either the Prefect Discourse forum or the Prefect Slack community.

Feel free to ⭐️ or watch prefect-firebolt for updates too!

Development

If you'd like to install a version of prefect-firebolt for development, clone the repository and perform an editable install with pip:

git clone https://github.com/PrefectHQ/prefect-firebolt.git

cd prefect-firebolt/

pip install -e ".[dev]"

# Install linting pre-commit hooks
pre-commit install

prefect-firebolt's People

Contributors

Stargazers

Watchers

prefect-firebolt's Issues

Create `FireboltDatabase` block

Create block that allows users to store configuration for a Firebolt database and retrieve and authenticated connection to the configured Firebolt database.

Add task for stopping a Firebolt engine

Firebolt engines incur cost based on how long their running. Firebolt offers the ability to stop an engine via their SDK. We should create a task that allows users to easily stop their Firebolt engine within their Prefect flows after running queries to help save on costs associated with running an engine.

I'd expect the interface to look something like this:

def stop_firebolt_engine(credentials: FireboltCredentials, engine_name: str):
    ....

Add task to retrieve a Firebolt engine's status

Since a Firebolt engine needs to be running in order run queries, we'll want the ability to check that an engine is running before running a query. This will also be useful when waiting for an engine that has been started to be available for use. Firebolt offers the ability to locate an engine via their SDK. The returned engine object can be used to get the status of the engine.

I'd expect the interface to look something like this:

def get_firebolt_engine(credentials: FireboltCredentials, engine_name: str) -> Engine:
    ....

query_firebolt raises error on insert/update/delete statements

Reporting on behalf of a user. When running any sort of modification queries (insert/update/delete), the query_firebolt task raises an error when cursor.fetchall() is called because there are no results to fetch. We could include a second task (execute or similar) that doesn't call fetchall. A further enhancement would be to return the values affected via cursor.rowcount (or even the cursor itself) on the new task.

Add task for starting a Firebolt engine

Firebolt engines need to be started before they can be used to execute queries. Firebolt offers the ability to start an engine via their SDK. We should create a task that allows users to easily start their Firebolt engine within their Prefect flows before running queries to increase the reliability of flows that use Firebolt.

I'd expect the interface to look something like this:

def start_firebolt_engine(credentials: FireboltCredentials, engine_name: str):
    ....

Create `query_firebolt` task

Create a task that accepts a FireboltDatabase block that submit a query against the configured database and returns the results.

prefecthq / prefect-firebolt Goto Github PK

prefect-firebolt's Introduction

prefect-firebolt

Welcome!

Getting Started

Python setup

Installation

Write and run a flow

Resources

Development

prefect-firebolt's People

Contributors

Stargazers

Watchers

prefect-firebolt's Issues

Recommend Projects

Recommend Topics

Recommend Org