paulescu / hands-on-train-and-deploy-ml Goto Github PK

View Code? Open in Web Editor NEW

672.0 672.0 111.0 18.76 MB

Train and Deploy an ML REST API to predict crypto prices, in 10 steps

Home Page: https://www.realworldml.net/subscribe

License: MIT License

Makefile 3.09% Python 95.45% Shell 1.45%

crypto deployment ml mlops

hands-on-train-and-deploy-ml's Introduction

Wanna become a Machine Learning PRO?

Hi there! 👋

I am Pau, the one with the beard 🧔🏻, the little one is my son Kai.

I am a mathematician who loves learning new things and solving problems. And this whole new world is full of them.

But what is learning without sharing? To me, not much 😊. Hence, datamachines.

For the last 4 years, I’ve been working as a freelancer on different Machine Learning projects, and the journey’s been inspiring.

So, I decided to start with a course on Reinforcement Learning, and so far I have an impression it’s doing well.

The idea is to share what I’ve learned and give value to the community.

So, thank you guys for following, it makes me very happy.

Stick around and get in touch, I am always up for new ideas and problems.

If you want to stay up to date with my latest courses and blogs:

👉🏽 Subscribe to the datamachines newsletter.

👉🏽 Follow me on Medium.

Keep on learning 🧠❤️

hands-on-train-and-deploy-ml's People

Contributors

Stargazers

Watchers

Forkers

andysingal iheb-bibani sriraj22 eliekawerk niskarsh12 gkakavel willytsinda trungducnguyenvn amircp mz0in m0saan ejmpa kayjayi chemelaa abdoiiii adrianmanresa pauljw28 pandyaved98 solab5 saikirankondapalli03 diegocotacio historyliao jordilucas16 code4indo chris-22ozor martinrojo1 dconstan wallisonferreira hdnminh taltaf913 9trahitariei hiroforyou jesusoctavioas mgrillo75 nitingithub seshakiran k-minutti rahulnanellu tomchapin techthiyanes dst1213 cheptoo-tk mayurlohana patelpooja1998 jonolummy abhishek085 acmoudleysa hirajanwin touristshaun roh1702 mussewold vinsy2774 baha-arfaoui mjndai7 lucasmellon-ops bastinrobin obinnaobeleagu fiyinfoluwa6 samkj brianconn71 sandyfit mats-dodd greviansonula jaedukseo ogsiji ab2021 rafaeljesus hazendeveloper majiouk restevesd vck luissalazarsalinas mehmetcanbudak lgs chukwukwadorom sameer-ahmadd coding4vinayak sorokinvld victormikhaylov jshuadvd kwang-le jorayala petreli12 bhupenderbindal lpai-org f0lk3n pranith7 taiwothomas doctorbinsibar vishnusharma03 victor-hro ryanlisse praisendebele hsiehchou folu22 productinfo jonatasamorim carledren inschools-ng brunogupa

hands-on-train-and-deploy-ml's Issues

`make init` not liking lightgbm version.

Doing a make init gives me this error.

$ make init 
curl -sSL https://install.python-poetry.org | python3 -
Retrieving Poetry metadata

The latest version (1.7.1) is already installed.
poetry install
Updating dependencies
Resolving dependencies... (2.9s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing lightgbm (3.3.5): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  INFO:root:running bdist_wheel
  INFO:root:running build
  INFO:root:running build_py
  INFO:root:creating build
  INFO:root:creating build/lib
  INFO:root:creating build/lib/lightgbm
  INFO:root:copying lightgbm/callback.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/compat.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/plotting.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/__init__.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/engine.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/dask.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/basic.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/libpath.py -> build/lib/lightgbm
  INFO:root:copying lightgbm/sklearn.py -> build/lib/lightgbm
  INFO:root:running egg_info
  INFO:root:writing lightgbm.egg-info/PKG-INFO
  INFO:root:writing dependency_links to lightgbm.egg-info/dependency_links.txt
  INFO:root:writing requirements to lightgbm.egg-info/requires.txt
  INFO:root:writing top-level names to lightgbm.egg-info/top_level.txt
  INFO:root:reading manifest file 'lightgbm.egg-info/SOURCES.txt'
  INFO:root:reading manifest template 'MANIFEST.in'
  WARNING:root:no previously-included directories found matching 'build'
  WARNING:root:warning: no files found matching '*.so' under directory 'lightgbm'
  WARNING:root:warning: no files found matching '*.so' under directory 'compile'
  WARNING:root:warning: no files found matching '*.dll' under directory 'compile/Release'
  WARNING:root:warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
  WARNING:root:warning: no previously-included files matching '*.py[co]' found anywhere in distribution
  WARNING:root:warning: no previously-included files found matching 'compile/external_libs/compute/.git'
  INFO:root:adding license file 'LICENSE'
  INFO:root:writing manifest file 'lightgbm.egg-info/SOURCES.txt'
  INFO:root:copying lightgbm/VERSION.txt -> build/lib/lightgbm
  INFO:wheel:installing to build/bdist.macosx-11.0-arm64/wheel
  INFO:root:running install
  INFO:LightGBM:Starting to compile the library.
  INFO:LightGBM:Starting to compile with CMake.
  Traceback (most recent call last):
    File "<string>", line 95, in silent_call
    File "/Users/nbhandari/mambaforge/envs/manother_ml_project_env/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make', '_lightgbm', '-I/private/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpef06d3aq/lightgbm-3.3.5/build_cpp', '-j4']' returned non-zero exit status 2.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/nbhandari/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 334, in <module>
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 403, in run
      self.run_command("install")
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/var/folders/gb/xd14qbk12bs52pkzwdsx3qlm0000gp/T/tmpj7eakzvm/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "<string>", line 248, in run
    File "<string>", line 199, in compile_cpp
    File "<string>", line 99, in silent_call
  Exception: An error has occurred while building lightgbm library file
  The full version of error log was saved into /Users/nbhandari/LightGBM_compilation.log
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with lightgbm (3.3.5) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "lightgbm (==3.3.5)"'.

make: *** [init] Error 1

Using mamba as virtual environment. Here are the details:

$ mamba info                            

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


     active environment : manother_ml_project_env
    active env location : /Users/nbhandari/mambaforge/envs/manother_ml_project_env
            shell level : 1
       user config file : /Users/nbhandari/.condarc
 populated config files : /Users/nbhandari/mambaforge/.condarc
                          /Users/nbhandari/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.1.2=0
                          __unix=0=0
       base environment : /Users/nbhandari/mambaforge  (writable)
      conda av data dir : /Users/nbhandari/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/nbhandari/mambaforge/pkgs
                          /Users/nbhandari/.conda/pkgs
       envs directories : /Users/nbhandari/mambaforge/envs
                          /Users/nbhandari/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Darwin/23.1.0 OSX/14.1.2
                UID:GID : 502:20
             netrc file : /Users/nbhandari/.netrc
           offline mode : False

Is this abandoned?

There's nothing in the Github actions section.

ModuleNotFoundError during deploy

Dear @Paulescu ,

I have been working on a personal project with the goal of understanding and implementing some of the most relevant concepts on Machine Learning Operations - MLOps. This personal projects is being developed using this repository as inspiration and guideline.

I am currently finishing this project and everything works fine except for the last step - deployment (using cerebrium). During the running of the Makefile - which "implements" deploy - I am facing a ModuleNotFoundError that I do not understand, which follows bellow.

As mentioned, I do not understand this error, since the structure of the deployment directory is as follows:

Additionaly, I provide my Makefile, so you have more means to understand why I am facing this situation, which besides lots of experimentation I do not understand. I honestly thank you for your time in helping me solve this error and also for the effort of putting together, and free of charge, this project as well all related materials.

Thank you.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.