Code Monkey home page Code Monkey logo

mlops_fuzzy-fish-waffle's Introduction

mlops_fuzzy-fish-waffle

The dataset is found on kaggle called "A Large Scale Fish Dataset". We want to utilize kornia to do image augmentation and to train the network. The model will be a simple convolutional neural network to classify the 9 different kind of fish.

The augmentation of the images will be to crop the images to focus on the center. To flip the images and rotate them since pictures of fish could potentially be in all directions. The images will also be normalized.

The model will have 3 convolutional layers with batchnormalization and relu will be used as activation function. The images will be flatten and fed into a dense hidden layer that continues into the classification layer. This will hopefully give an accuracy higher than 97%. We will start by creating this model using pytorch. If there is more time a model will be created using kornia.

Instructions for torch serve

If folder "model_store" is empty RUN:

torch-model-archiver --model-name my_fancy_model --version 1.0 --serialized-file models/deployable_model.pt --export-path model_store --extra-files torchserve/index_to_name.json --handler torchserve/myImageClass.py

Otherwise just start torchserve

  1. torchserve --start --ncs --model-store model_store --models my_fancy_model=my_fancy_model.mar
  2. In another terminal run: curl http://127.0.0.1:8080/predictions/my_fancy_model -T im00.png

Week 1

  • Create a git repository
  • Make sure that all team members have write access to the github repository
  • Create a dedicated environment for you project to keep track of your packages (using conda)
  • Create the initial file structure using cookiecutter
  • Fill out the make_dataset.py file such that it downloads whatever data you need and
  • Add a model file and a training script and get that running
  • Remember to fill out the requirements.txt file with whatever dependencies that you are using
  • Remember to comply with good coding practices (pep8) while doing the project
  • Do a bit of code typing and remember to document essential parts of your code
  • Setup version control for your data or part of your data
  • Construct one or multiple docker files for your code
  • Build the docker files locally and make sure they work as intended
  • Write one or multiple configurations files for your experiments
  • Used Hydra to load the configurations and manage your hyperparameters
  • When you have something that works somewhat, remember at some point to to some profiling and see if you can optimize your code
  • Use wandb to log training progress and other important metrics/artifacts in your code
  • Use pytorch-lightning (if applicable) to reduce the amount of boilerplate in your code

Week 2

  • Write unit tests related to the data part of your code
  • Write unit tests related to model construction
  • Calculate the coverage.
  • Get some continuous integration running on the github repository
  • (optional) Create a new project on gcp and invite all group members to it
  • Create a data storage on gcp for you data
  • Create a trigger workflow for automatically building your docker images
  • Get your model training on gcp
  • Play around with distributed data loading
  • (optional) Play around with distributed model training
  • Play around with quantization and compilation for you trained models

Week 3

  • Deployed your model locally using TorchServe
  • Checked how robust your model is towards data drifting
  • Deployed your model using gcp
  • Monitored the system of your deployed model
  • Monitored the performance of your deployed model

mlops_fuzzy-fish-waffle's People

Contributors

duritajonsdottir avatar granatb avatar marcuskbrandt avatar wojciechtyczynski avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.