Importing a new package that doesn't exist in `Pyodide` about pyscript HOT 20 CLOSED

hi, i am excited to see your interest in this project. :)

i have been working on getting torch into Pyodide. it's pretty challenging.

we do have a alpha workflow to get non-pure python packages with c/c++/cython extensions into the pyscript/Pyodide ecosystem, but torch is extra complicated.

i plan on posting the build process for non-pure python packages as soon as i can

from pyscript.

Even more exciting from my perspective would be JAX. Are you working on this in a public repo?

There was a discussion about JAX in pyodide/pyodide#2198, tensorflow in pyodide/pyodide#50 and torch in pyodide/pyodide#1625.
It would be great to have them but they are all fairly large and complex packages, so it would take a lot of work to make that happen.

I would suggest you take a look at onnx

For ML wrapping JS libraries (such as ONNX Runtime Web, Tensorflow.js etc) from Python via the Pyodide's FFI is indeed another solution. Though your API is then not going to be exactly the same as when using e.g. Tensorflow from Python, so one cannot reuse code without changes.

from pyscript.

currently, you should be able to use all the packages here

https://github.com/pyodide/pyodide/tree/main/packages

you can build your own packages and add them into pyscript. but some packages are very difficult to build. torch and tensorflow are hard to build into WASM and the WASM compiler tooling. i am working on porting torch now and tensorflow after that. i hope to make some new posts as i progress on this task.

from pyscript.

@verhulstm I'm super excited to hear about the progress you make on torch and tensorflow. Even more exciting from my perspective would be JAX. Are you working on this in a public repo? I'd love to follow along and help where I can.

from pyscript.

I would suggest you take a look at onnx

Yep, ONNX web runtime, tfjs-tflite, etc. are fine for inference - but JAX/torch/tensorflow can do much more than model inference. It would be a dream come true to be able to play with JAX's auto-differentiation in the browser!

from pyscript.

Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?

Packages that don't have pure Python wheels,

• require to run setup.py with setuptools which currently doesn't work due to unsupported features in the browser pyodide/pyodide#1501
• even if it setuptools could be made to work, there is no guarantee that setup.py would run without error. At least with pure Python wheels it if exists, there is a guarantee that it would be installed correctly.
• finally packages with binary extensions require to be built with Emscripen (currently using the Pyodide build system) and so those cannot be installed in the browser from .tar.gz in any case.

We have added a FAQ entry recently on what to do when a package fails to install https://pyodide.org/en/latest/usage/faq.html#micropip-can-t-find-a-pure-python-wheel

it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can't we store already installed packages to load them?

Yes, they will be re-installed each time. However the browser would cache them, so they don't have to be re-downloaded and it's mostly a matter of unzipping them to the virtual file system (which is fast) and pre-loading dynamic libraries contained in the package (see below).

If the packages are already included in Pyodide, it will only load them and not fetch them from PyPI and install them.

Yes, in particular for packages included in Pyodide we unvendor tests, so they are a bit smaller. Otherwise, for pure Python packages, there is little difference from where to install them.

Why PyScript is super slow? I am assuming this because of the installation of packages from scratch

If you profile a typical Python application say that loads pandas results would be roughly the following:

• loading CPython in Pyodide: 25%
• download the package files: depends on your network connexion but it's usually not the bottleneck
• unpack package files to the virtual file system: 3%
• pre-load dynamic libraries: 55%
• import pandas: 15%

A large part of the time would be spent compiling .wasm code objects by the browser. This situation is made worse by the fact that we cannot load dynamic libraries synchronously on demand when Python modules are imported and have to-preload/compile all the libraries in a package when the package is loaded. Libraries such as numpy, pandas etc have a lot of binary extensions which take a while to compile (the same goes for CPython itself). Some browsers would cache WASM compilation https://v8.dev/blog/wasm-code-caching but it's a step that takes a while the first time. There is active work happening on how to reduce load time in Pyodide but it's still a challenging topic.

from pyscript.

Joining late to the party... a bunch of the questions in the original post was already addressed so I'll focus on the ones that I think were left out:

• How can I add a newer version of Pyodide which was built locally to Pyscript?

Currently, you can't. Opened #172 to address this

• Isn't there a way to import a python environment that has all of the packages already installed?

I think the question here means to load a packaged "bundle" with pyodide and all (and only) applications require packages once. Right? If so, no, this is not possible today. This is actually an upstream issue that we recently discussed with the pyodide team and we have some ideas on.. but not in the short term. /cc @rth

• Why do we need to have pure Python 3 wheel (aka needs to be compiled on all hardware)? Is it because it is actually being compiled on the browser-side which should be able to run all hardware?

Not entirely sure I understand question but, in any way I could interpret it, it is an upstream question. @rth @hoodmane do you want to chime in?

From looking into the logs, it seems that every time the page has loaded the packages listed to be used get installed by micropip, is this correct? If yes, why can't we store already installed packages to load them?

If the packages are already included in Pyodide, it will only load them and not fetch them from PyPI and install them. I'd think it's to optimize size and performance (the more you load/add the bigger and slower it is).

Why PyScript is super slow? I am assuming this because of the installation of packages from scratch (if my assumption is true).

We are trying to investigate in the next few days but it's very very likely to be an upstream issue.

Thank you for the questions! As we roll out better documentation, we'll add some of these to the FAQ. 👍

from pyscript.

closing as upstream, seems that questions have been answered and there's not much else to do. Happy to be convinced otherwise :)

from pyscript.

if you have a pure python file that should already work. include a link to the file in your "py-env" tag

from pyscript.

Thanks @verhulstm and @sudomaze!

from pyscript.

great, i would love to chat in the next few days. email me :)

from pyscript.

hi, what about tensorflow as it has the javacript package in it. will it be able to work with pyscript.

from pyscript.

i see more clearly now, sorry. i expect TensorFlow.js will work yes.

good idea. i will try that out

from pyscript.

@verhulstm I would suggest you take a look at onnx. They have a JavaScript library for running PyTorch models into browser.

https://cloudblogs.microsoft.com/opensource/2021/09/02/onnx-runtime-web-running-your-machine-learning-model-in-browser/

from pyscript.

you can build your own packages and add them into pyscript.

@verhulstm Are there any examples of how you would do this? I'd love to experiment with using this with some pure python libraries that aren't on pypi.

I'm assuming you'd follow the steps on how to build a Pyodide package but I'm not clear on how you'd add them into pyscript.

Thanks!

from pyscript.

i am prepping a better guide for this, but until that is ready.

get a pyodide dev env ready. in docker or vagrant

git clone https://github.com/pyodide/pyodide.git
git checkout 0.20.0a1
source pyodide_env.sh

then follow https://pyodide.org/en/stable/development/new-packages.html

from pyscript.

i did this in ubuntu 20.04

with these packages

apt install -y build-essential
apt install -y linux-headers-$(uname -r)
apt install -y libgdbm6 libdbus-glib-1-2 libtool
apt install -y libssl-dev libyaml-dev libreadline6-dev libncurses5-dev libffi-dev libgdbm-dev libdb-dev libltdl-dev
apt install -y zlib1g-dev
apt install -y clang-format-6.0
apt install -y prelink cmake patch automake autoconf bzip2 bison ccache pkg-config swig xz-utils autotools-dev
apt install -y texinfo dejagnu gnupg2
apt install -y git
apt install -y unzip
apt install -y wget
apt install -y gfortran
apt install -y g++
apt install -y f2c
apt install -y default-jre
apt install -y nodejs
apt install -y npm

and

https://repo.anaconda.com/miniconda/Miniconda3-py39_4.11.0-Linux-x86_64.sh

from pyscript.

You can link pure Python 3 wheels directly in Pyscript without the need of the package to be in Pyodide. I have tested with different packages where I have found pure Python 3 wheels.

import asyncio
import micropip

await micropip.install(URL)

I don't know if py-env supports external URLs.

from pyscript.

@verhulstm let me know if I can help, I will be happy to contribute.

The PyTorch team is aware of your contribution to import PyTorch into Pyodide, so they should be in contact with you

cc: @msaroufim

from pyscript.

Would you like to ask if it supports loading torch models now？

from pyscript.