prefect-patterns's Introduction

prefect-patterns

How to Run Example Flows from this Repo

Fork or Clone this Repository

Run the following commands from the root level of your repository.

Create Virtual Environment

conda create --name prfp python=3.9

conda activate prfp

Install Dependencies

pip install -U prefect 
pip install prefect-aws prefect-docker prefect-github python-dotenv

Authenticate to Prefect Cloud and Set Workspace

prefect cloud login
prefect cloud workspace set

Set Up AWS Creds and S3 Bucket

Add a .env file with the following variables:

AWS_ACCESS_KEY_ID=add-your-key-id
AWS_SECRET_ACCESS_KEY=add-your-secret-access-key

Create an S3 Bucket with name se-demo-result-storage.

Create Blocks

python utilities/blocks.py

Create Local Docker Work Pool

prefect work-pool create local-docker --type docker

Reconfigure Build and Pull Steps where Necessary

In the prefect.yaml file:

1. Build Step: should point to your image registry.

build:
- prefect_docker.deployments.steps.build_docker_image:
    requires: prefect-docker>=0.3.0
    image_name: taycurran/test-projects-june11 # CHANGE HERE
    tag: '{{ get-commit-hash.stdout }}'
    dockerfile: auto
    push: true

2. Pull Step: If you forked this repo, point to your github URL.

pull:
- prefect.deployments.steps.git_clone:
    repository: https://github.com/taylor-curran/prefect-patterns.git # CHANGE HERE
    branch: main

Create Deployments

prefect deploy --all

Start Worker

Ensure you have a docker damon running.
Open up a new terminal and start your docker typed worker:

prefect worker start --pool local-docker

prefect-patterns's People

Contributors

Stargazers

Watchers

prefect-patterns's Issues

Bug Template

First check

I added a descriptive title to this issue.
I used the GitHub search to find a similar issue and didn't find it.
I searched the Prefect documentation for this issue.
I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Reproduction

Error

Versions

Additional context

UI Template

First check

I added a descriptive title to this issue.
I used the GitHub search to find a similar issue and didn't find it.
I refreshed the page and this issue still occurred.
I checked if this issue was specific to the browser I was using by testing with a different browser.