Code Monkey home page Code Monkey logo

workshop-e2e-gitops-ml's Introduction

Workshop - End2End GitOps ML pipeline

This is a Computer Vision (CV) project that solves the problem of segmenting out swimming pools from satellite images.

We use a slightly modified version of the BH-Pools dataset: we split the original 4k images into tiles of 1024x1024 pixels.

Initial setup

Python 3.8+ is required to run code from this repo.

  1. Send the presenter the e-mail address associated with your GitHub account. You will be invited to collaborate on the workshop repository.

  2. Clone the repository locally

    $ git clone https://github.com/tibor-mach/workshop-e2e-gitops-ml
    $ cd workshop-e2e-gitops-ml
  3. Now let's install the requirements and create a virual environment. You can use the included makefile and simply call

    $ make venv
  4. (Optional) Connect your account to DVC Studio for live experiment sharing

    The worskhop host should invite you to a team in DVC Studio using the GitHub-associated e-mail address. You can then simply use your GitHub account to log in to Studio.

    Once you are signed-in, click on the icon in the top-right corner of the Studio UI and select Profile. Scroll down to Studio Access Token and copy it

    Then call the following command with *** replaced by your token

    dvc config --global studio.token ***

    This will allow dvc to communicate with Studio and share progress of live (ongoing) experiments with others.

1. Getting raw data

Import the raw data with dvc from our public dataset registry:

$ dvc import https://github.com/iterative/dataset-registry use-cases/pool_data -o data/pool_data

1. Running your own experiments during the workshop

  1. Open the params.yaml file in the root of the repository and edit some values
  2. Run dvc exp run
  3. Observe the results in Studio, in the VS code extension or in the command line
  4. Persist the experiment and add it to the (local and remote) git history by calling
    dvc exp push origin exp-name
    with exp-name replaced with your experiment name.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.