fastai / fastconda Goto Github PK

View Code? Open in Web Editor NEW

25.0 2.0 9.0 1.19 MB

Get packages onto your conda channel faster

License: Apache License 2.0

Shell 2.45% Python 26.64% Batchfile 0.67% Jupyter Notebook 70.24%

fastconda's Introduction

fastconda

Get packages onto your conda channel faster

Status

Builds

Package Name	Pypi	GitHub	Conda fastai channel
sentencepiece
opencv-python-headless		N/A
timm
albumentations
imgaug

Copies

Package	Pypi	Conda fastchan channel
cudf
cudatoolkit	N/A
mamba
pytorch
torchvision
transformers
rich
sentencepiece
fastai
timm
nbdev
fastrelease
ghapi
fastcgi

Build Process

We are using three different ways for sourcing a Conda package into an Anaconda repo:

conda build: When there are C dependencies.
setuptools-conda: For pure python packages.
anaconda copy: When a maintained Anaconda package already exists.

conda build

When there are C dependencies.

Build a Conda package by first installing the appropriate pip package(s) in a fresh Conda environment, and then use conda build to build a package based on this environment. We do this for packages that have C dependencies and need thus need binaries created for different platforms. This build process is specified in condabuild.yml. This type of build requires a specific directory structure with several metadata files which amounts to a fair amount of boilerplate. For this reason, we dynamically generate all of this boilerplate based on the configuration file build.yaml

The schema of build.yaml is as follows:

- pypinm: opencv-python-headless
  import_nm: cv2 # Optional: if not specified will default to the same as pypinm
  deps: [numpy]  # Optional: if not specified will just be python
  path: destination_dir # Optional: if not specified, files will be placed in a directory named after `pypinm`
- pypinm: sentencepiece # This second package, `sentencepiece`, uses all of the defaults.

In the above example, we specify two packages to built, opencv-python-headless and sentencepiece. Here is a description of all fields:

pypinm: this field is required and is the name of the package on pypi.
import_nm: You only need to supply this field when the import name of the pypi package is different than the package name. For example the import name of opencv-python-headless is cv2.
deps: This is a list of all dependencies. If this is not supplied, no dependencies beyond python are assumed.
path: You should usually ignore this field completely and rely on the default behavior. This optional parameter allows you to specify the directory where the metadata files will be placed. If not specified, files will be placed in a directory named after pypinm.

You can run this locally with:

python build.py

see condabuild.yml for necessary environment setup.

setuptools-conda

For pure python packages.

For python packages that are pure-python that do not require binaries, we can instead create a cross-platform Conda package using setuptools-conda. This build process is specified in setupconda.yaml.

You can run this locally with:

./setupconda.sh {args}

see setupconda.yaml for example of args

anaconda copy

When a maintained Anaconda package already exists.

In situations where there is a reliable and maintained conda package already present in another channel, we can copy this package and all its dependencies to another channel. This is desirable when you want to simplify and speed up the installation of packages by placing all dependencies in a single channel. This process is carried out via anacopy.yml. We find all dependencies for a particular package by doing a Conda installation, which uses the Conda solver to find all the dependencies with appropriate version numbers, and then copy the appropriate packages using anaconda copy.

You can run this locally:

python get_deps.py

See anacopy.yml for the full workflow.

fastconda's People

Contributors

Stargazers

Watchers

Forkers

ankitshah009 rakhithjk muellerzr isabella232 ralphbrodriguez christofkaufmann azbab-eng sethfri nyctophile-aayu

fastconda's Issues

Simplify `anaconda copy` with dependencies

For each python version:

conda create -yn tmp python==3.9
conda activate tmp
conda install -c defaults -c conda-forge -d sentencepiece transformers rich mamba timm --json > inst.json

Then copy the conda-forge ones over to fastai channel.

Fastchan::sympy 404 not found

I cannot build my docker image because a link used to install the sympy package leads to a 404 page.

Keep-Alive-Actions

setupconda and condabuild have been failing for 3 months

Hi @jph00! It looks like the setupconda and condabuild workflows have been failing for ~3 months. The last time I can see them running successfully consistently is 20 pages back in the workflow runs.

I believe this might be why I'm seeing the issue I'm seeing in #37 where fastchan doesn't have the latest version of platformdirs and fails fastai installation.

Do you mind taking a look at getting the workflows running again? Thanks for all you do! The fast.ai course has been a great help so far

Keep-Alive-Actions

Create module and workflow to create conda packages from pypi

For situations where there isn't a conda-forge package, or it's not up-to-date, we can create our own package.

This could use grayskull, or its online version:

https://marcelotrevisani.info:8000/recipe/?name=grayskull

It could also use setuptools-conda.

Keep-Alive-Actions

opencv-python-headless conda package for python > 3.11

Hope this is the proper place to ask this question.

I tried to install the opencv-python-headless on a conda environment with python3.12

>> conda create -n test -c fastai python=3.12 opencv-python-headless
Retrieving notices: done
Channels:
 - fastai
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides _python_rc needed by python-3.12.0rc3-rc3_hab00c5b_1_cpython

Could not solve for environment specs
The following packages are incompatible
├─ opencv-python-headless is installable with the potential options
│  ├─ opencv-python-headless 4.5.1.48 would require
│  │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  ├─ opencv-python-headless [4.10.0.82|4.10.0.84|...|4.9.0.80] would require
│  │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│  ├─ opencv-python-headless 4.5.1.48 would require
│  │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│  ├─ opencv-python-headless [4.10.0.82|4.10.0.84|...|4.9.0.80] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ opencv-python-headless [4.5.2.52|4.5.2.54|4.5.3.56|4.5.4.58|4.5.4.60] would require
│  │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│  ├─ opencv-python-headless [4.10.0.82|4.10.0.84|...|4.9.0.80] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  ├─ opencv-python-headless [4.10.0.82|4.10.0.84|...|4.9.0.80] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  └─ opencv-python-headless [4.10.0.82|4.10.0.84|4.9.0.80] would require
│     └─ python >=3.11,<3.12.0a0 , which can be installed;
└─ python 3.12**  is not installable because there are no viable options
   ├─ python [3.12.0|3.12.1|...|3.12.8] would require
   │  └─ python_abi 3.12.* *_cp312, which conflicts with any installable versions previously reported;
   └─ python 3.12.0rc3 would require
      └─ _python_rc, which does not exist (perhaps a missing channel).

Checking https://anaconda.org/fastai/opencv-python-headless it seems like there are no packages for 3.12 (and for 3.13)

Is it possible to add them?

Keep-Alive-Actions

Workflows have not run in 4 months

Hi all,

the workflows haven't run in 4 months because there was no activity in this repo over the past 60 days.

Could you enable them again? 🙏🏻

Keep-Alive-Actions

fastchan is missing platformdirs 3.10.0

I tried to run mamba install -c fastchan fastai but it fails, hitting a 404 error in trying to download platformdirs version 3.10.0 (error below). Looking at Anaconda, it looks like the latest platformdirs version in fastchan is indeed 3.9.1 instead of 3.10.0, but it is 3.11.0 in conda-forge.

Multi-download failed. Reason: Transfer finalized, status: 404 [https://conda.anaconda.org/fastchan/noarch/platformdirs-3.10.0-pyhd8ed1ab_0.conda] 17 bytes

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/mamba/mamba.py", line 941, in exception_converter
        raise e
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/mamba/mamba.py", line 934, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/mamba/mamba.py", line 892, in _wrapped_main
        result = do_call(parsed_args, p)
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/mamba/mamba.py", line 754, in do_call
        exit_code = install(args, parser, "install")
      File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/mamba/mamba.py", line 588, in install
        transaction.fetch_extract_packages()
    RuntimeError: Multi-download failed. Reason: Transfer finalized, status: 404 [https://conda.anaconda.org/fastchan/noarch/platformdirs-3.10.0-pyhd8ed1ab_0.conda] 17 bytes

Keep-Alive-Actions

Automate creation of conda build assets

Problem: Conda builds require a directory with files:

Example: https://github.com/fastai/fastconda/tree/master/opencv-python-headless

build.bat
meata.yaml
build.sh

You don't have to understand anything about these files, except that these are basically boilerplate with a few variables of things that need to be changed:

bld.bat: example in this file https://github.com/fastai/fastconda/blob/master/opencv-python-headless/bld.bat you need to replace package name with a variable that is the package name
build.sh also just need to replace the package name, however this one can be done via an environment variable called PKG_NM But Jeremy will change this detail. Reference: https://docs.conda.io/projects/conda-build/en/latest/user-guide/environment-variables.html#id2
In Meta.yaml
1. Need to fill in the name and version (see the template variables and this will make sense)
2. Need to fill in the requirements.run list of dependencies, which can be supplied via arguments from the user
3. Need to fill in the test.imports. section with an import name, which can be different than the package name (example: opencv-python-headless vs cv2) -- investigate this to see if you can get the import name automatically somehow or if it needs to be supplied by the user.

Approach: look at fastrelase

pypi_details
update_meta
write_pip_conda_meta for an example of how to create yaml files

Things I'm not 100% sure about but will investigate:

Can I get the import name automatically ex: cv2 vs opencv-python-headless
Does the file that I materialize need to to be meta.yaml not meta.yaml.tmpl correct? I assume that meta.yaml.tmpl is an intermediate file that is processed outside of conda build but maybe conda build is providing direct support for templating?

@jph00 please review LMK if I'm missing anything.

fastai / fastconda Goto Github PK

fastconda's Introduction

fastconda

Status

Builds

Copies

Build Process

conda build

setuptools-conda

anaconda copy

fastconda's People

Contributors

Stargazers

Watchers

Forkers

fastconda's Issues

Problem: Conda builds require a directory with files:

Recommend Projects

Recommend Topics

Recommend Org