This repository contains the tools needed to build the Open-CE project.
The open-ce
tool allows a user to build collections of conda recipes described within a collection of feedstocks. It also provides tools for validating potential conda environments and running tests for feedstocks.
conda
>= 3.8.3conda-build
>= 3.20- Once
conda
is installed,conda-build
can be installed with the command:conda install conda-build
- Once
networkx
>= 2.5python
>= 3.6docker
>= 1.13 orpodman
>= 2.0.5- docker or podman required only when building within a container (see below).
Currently CUDA 10.2 and 11.0 are supported by the recipes in Open-CE. Please see doc/README.cuda_support.md
for details on setting
up a proper build enviornment for CUDA support.
Open-CE expects the CUDA_HOME
environment variable to be set to the location of the CUDA installation. Note that not all recipes work when CUDA_HOME
references a non-standard CUDA installation location. Reference the cuda README for more information.
When building packages that use CUDA, a tar package of TensorRT for the intended CUDA version will need to be downloaded ahead of time. The downloaded file should be placed in a new local directory called local_files
. The cuda README has more information.
To get the Open-CE build tools, one can either install them via conda
from the Open-CE channel, install them via pip
from github or clone the source code from github as below -
# Conda install from the open-ce channel
conda install -c open-ce open-ce-builder
OR
# Pip install from the main branch
pip install git+https://github.com/open-ce/open-ce-builder.git@main
OR
# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce-builder.git
cd open-ce-builder
pip install -e .
To build an entire integrated and functional conda channel using Open-CE, start by installing the needed tools in the Requirements section above.
The open-ce build env
command can then be used to build a collection of Open-CE packages. An Open-CE environment file needs to be passed in as input. A selection of environment files are provided within the open-ce
repo for different frameworks such as TensorFlow and PyTorch. The output from running open-ce build env
will be a local conda channel (by default called condabuild
) and one or more conda environment file(s) in the output folder depending on the selected build configuration. For more details on open-ce build env
, please see doc/README.open_ce_build.md
.
The following commands will use the opence-env.yaml
Open-CE environment file to build all of the Open-CE packages for Python 3.7 (the default), including CUDA builds and cpu-only builds (also the default). The commands should be run from within the same directory that contains local_files
.
# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce.git
# Build packages
open-ce build env ./open-ce/envs/opence-env.yaml
The open-ce
tool will also automatically look for environment files within the open-ce repo's env directory if an environment file isn't found locally.
The following commands will build the opence-env.yaml
environment file:
# Build packages
open-ce build env opence-env
A specific version of an environment file from the open-ce repo can be built using the --git_tag
flag.
The following commands will build version 1.1.4 of the open-ce environment file provided within the open-ce repo:
# Build packages
open-ce build env --git_tag open-ce-v1.1.4 opence-env
The following commands will use the opence-env.yaml
Open-CE environment file from a specific Open-CE release to build all of the Open-CE packages for Python 3.6, 3.7 and 3.8, including only CUDA builds. The commands should be run from within the same directory that contains local_files
.
# Build packages
open-ce build env --python_versions 3.6,3.7,3.8 --build_types cuda opence-env
Passing the --container_build
argument to the open-ce build env
command will create a container image and perform the actual build inside of a container based on that image. This will provide a "clean" environment for the builds and make builds more system independent. It is recommended to build with this option as opposed to running on a bare metal machine. For more information on the --container_build
option, please see doc/README.open_ce_build.md
.
The open-ce build feedstock
command can be used to build a single feedstock (which could produce one or more conda packages). The output from running open-ce build feedstock
will be a local conda channel (by default called condabuild
). For more details on open-ce build feedstock
, please see doc/README.open_ce_build.md
.
The following commands will build all of the packages within a feedstock named MY_FEEDSTOCK
.
# Clone Open-CE Environments from GitHub
git clone https://github.com/open-ce/open-ce.git
# Clone MY_FEEDSTOCK from GitHub
git clone https://github.com/open-ce/MY_FEEDSTOCK-feedstock.git
# Build packages
cd MY_FEEDSTOCK-feedstock
open-ce build feedstock --conda_build_config ../open-ce/envs/conda_build_config.yaml
After performing a build, a local conda channel will be created. By default, this will be within a folder called condabuild
(it can be changed using the --output_folder
argument). After the build, packages can be installed within a conda environment from this local channel. If the packages are built using open-ce build env
script, then a conda environment file will also be generated which can be used to generate a conda environment with the built packages installed in it. See conda's documentation for more information on conda environments.
The following command will install a package named PACKAGE
from the local conda channel condabuild
into the currently active conda environment.
conda install -c ./condabuild PACKAGE
The following command can be used to create a conda environment using a conda environment file.
conda env create -f <conda_environment_file>
After performing the build using the open-ce build env
tool, the open-ce test
tool can be used to either test a package or a collection of packages. For more details on open-ce test
, please see doc/README.open_ce_test.md
.
After performing the build using open-ce build env
, the open-ce build image
command can be used to create a runtime container image containing the newly created conda channel, as well as a conda environment with the newly build Open-CE packages. For more details on open-ce build image
, please see doc/README.open_ce_build.md
.
For contribution information, please see the CONTRIBUTING.md page.