open-ce / open-ce-builder Goto Github PK

Build tools for Open-CE

License: Apache License 2.0

Python 96.71% Roff 1.21% Dockerfile 1.63% Shell 0.45%

open-ce-builder's Introduction

Open-CE Builder

This repository contains the tools needed to build the Open-CE project.

The open-ce tool allows a user to build collections of conda recipes described within a collection of feedstocks. It also provides tools for validating potential conda environments and running tests for feedstocks.

GETTING STARTED

Requirements

conda
- The conda tool can either be installed through Anaconda or Miniconda.
conda-build >= 3.22
- Once conda is installed, conda-build can be installed with the command: conda install conda-build
networkx >= 2.5
python >= 3.8
junit-xml >= 1.9
docker >= 1.13 or podman >= 2.0.5
- docker or podman required only when building within a container (see below).
matplotlib == 3.7.1
- Required only when exporting the dependency graph.

CUDA Requirements

Currently CUDA 11.8 and 12.2 are supported by the recipes in Open-CE. Please see doc/README.cuda_support.md for details on setting up a proper build enviornment for CUDA support.

Open-CE expects the CUDA_HOME environment variable to be set to the location of the CUDA installation. Note that not all recipes work when CUDA_HOME references a non-standard CUDA installation location. Reference the cuda README for more information.

When building packages that use CUDA, a tar package of TensorRT for the intended CUDA version will need to be downloaded ahead of time. The downloaded file should be placed in a new local directory called local_files. The cuda README has more information.

Installing the Open-CE Build Tools

To get the Open-CE build tools, one can either install them via conda from the Open-CE channel, install them via pip from github or clone the source code from github as below -

# Conda install from the open-ce channel
conda install -c open-ce open-ce-builder

# Pip install from the main branch
pip install git+https://github.com/open-ce/open-ce-builder.git@main

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce-builder.git
cd open-ce-builder
pip install -e .

Open-CE compatibility with Open-CE Builder

Open-CE version	Open-CE Builder version
1.7.10	12.0.1
1.8.1	12.0.1
1.9.4	12.0.3
1.10.0	12.0.3
1.11.0	13.0.1

Building a Collection of Packages

To build an entire integrated and functional conda channel using Open-CE, start by installing the needed tools in the Requirements section above. The open-ce build env command can then be used to build a collection of Open-CE packages. An Open-CE environment file needs to be passed in as input. A selection of environment files are provided within the open-ce repo for different frameworks such as TensorFlow and PyTorch. The output from running open-ce build env will be a local conda channel (by default called condabuild) and one or more conda environment file(s) in the output folder depending on the selected build configuration. For more details on open-ce build env, please see doc/README.open_ce_build.md.

The following commands will use the opence-env.yaml Open-CE environment file to build all of the Open-CE packages for Python 3.10 (the default), including CUDA builds and cpu-only builds (also the default). The commands should be run from within the same directory that contains local_files.

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce.git
# Build packages
open-ce build env ./open-ce/envs/opence-env.yaml

The open-ce tool will also automatically look for environment files within the open-ce repo's env directory if an environment file isn't found locally.

The following commands will build the opence-env.yaml environment file:

# Build packages
open-ce build env opence-env

A specific version of an environment file from the open-ce repo can be built using the --git_tag flag.

The following commands will build version 1.11.0 of the open-ce environment file provided within the open-ce repo:

# Build packages
open-ce build env --git_tag open-ce-v1.11.0 opence-env

The following commands will use the opence-env.yaml Open-CE environment file from a specific Open-CE release to build all of the Open-CE packages for Python 3.10 and 3.11, including only CUDA builds. The commands should be run from within the same directory that contains local_files.

# Build packages
open-ce build env --python_versions 3.10,3.11 --build_types cuda opence-env

Note that having conda-forge in your channel list may sometime cause conflicts or unexpected errors due to dependencies' versions mismatch. So, it is recommended to avoid mixing the channels during the build as well as at runtime.

Power10 MMA Optimization

Building Packages

One can build the major Open-CE libraries like TensorFlow, Pytorch, Xgboost, etc. with Power10 MMA optimization. For details, please see doc/README.open_ce_build.md.

Running Packages

These packages will work on Power9 or Power10, but not on Power8.

FFMPEG with reduced codecs

One can build Open-CE packages with ffmpeg in two ways:

Provide --build-ffmpeg option to the open-ce build env command. When this option is provided, Open-CE Builder builds envs/ffmpeg-env.yaml internally and then builds the provided Open-CE environment file for which the build is requested.

Building within a container

Passing the --container_build argument to the open-ce build env command will create a container image and perform the actual build inside of a container based on that image. This will provide a "clean" environment for the builds and make builds more system independent. It is recommended to build with this option as opposed to running on a bare metal machine. For more information on the --container_build option, please see doc/README.open_ce_build.md.

Building a Single Feedstock

The open-ce build feedstock command can be used to build a single feedstock (which could produce one or more conda packages). The output from running open-ce build feedstock will be a local conda channel (by default called condabuild). For more details on open-ce build feedstock, please see doc/README.open_ce_build.md.

The following commands will build all of the packages within a feedstock named MY_FEEDSTOCK.

# Clone Open-CE Environments from GitHub
git clone https://github.com/open-ce/open-ce.git
# Clone MY_FEEDSTOCK from GitHub
git clone https://github.com/open-ce/MY_FEEDSTOCK-feedstock.git
# Build packages
cd MY_FEEDSTOCK-feedstock
open-ce build feedstock --conda_build_config ../open-ce/envs/conda_build_config.yaml

Installing Packages

After performing a build, a local conda channel will be created. By default, this will be within a folder called condabuild (it can be changed using the --output_folder argument). After the build, packages can be installed within a conda environment from this local channel. If the packages are built using open-ce build env script, then a conda environment file will also be generated which can be used to generate a conda environment with the built packages installed in it. See conda's documentation for more information on conda environments.

The following command will install a package named PACKAGE from the local conda channel condabuild into the currently active conda environment.

conda install -c ./condabuild PACKAGE

The following command can be used to create a conda environment using a conda environment file.

conda env create -f <conda_environment_file>

Testing Packages

After performing the build using the open-ce build env tool, the open-ce test tool can be used to either test a package or a collection of packages. For more details on open-ce test, please see doc/README.open_ce_test.md.

Creating Container Image with Open-CE Packages installed

After performing the build using open-ce build env, the open-ce build image command can be used to create a runtime container image containing the newly created conda channel, as well as a conda environment with the newly build Open-CE packages. For more details on open-ce build image, please see doc/README.open_ce_build.md.

Contributions

For contribution information, please see the CONTRIBUTING.md page.

open-ce-builder's People

Contributors

Stargazers

Watchers

Forkers

bnemanich cdeepali dncliss bethune-bryant npanpaliya jayfurmanek sameer-d dllehr81 ketank-new siyaamonkar ritumd rocketsoftware mmitsugi archanashinde1 open-ce saisrivatsa smartvibs8876 aman-surkar archana-shinde1

open-ce-builder's Issues

[BUG] switch to using CLI commands instead of API

Looks like CLI commands work better compared to the APIs, so better to switch to using CLI commands in code. - conda/conda-build#4228 (comment)

Speed up performance of BuildTree

Creating a BuildTree now takes a significant amount of time due to the necessary changes in open-ce/open-ce#355. It currently takes a long time to call the conda search command. There are a couple of ideas to speed this up:

Provide the --use-index-cache argument while calling conda search.
Cache the results internally of previous calls to conda search, since there might be multiple calls for the same package.
Determine if there are any quicker ways to query conda.

Add CUDA 11.2 build images

Add CUDA 11.2 build images to the builder tool.

Allow `open-ce get licenses` to work with existing conda environments

It might be useful to let the open-ce get licenses command work with existing conda environments instead of only requiring a conda environment file. This would remove an extra step that a user might need to take of creating a conda environment file. A simple way of doing this would be to automatically generate a conda environment file when a user passes in a conda environment name and pass this to the open-ce get licenses command.

Skip building existing packages within a channel

Currently, open-ce build env will skip building a package if it already exists within the output_folder. This functionality should be extended to check if a package exists within all provided channels as well.

arviz test failure

[BUG] unable to build OpenCE 1.5.0 with docker

The following error is seen while building OpenCE 1.5.0 with docker:

HEAD is now at 841253d Release numactl 2.0.12
Traceback (most recent call last):
  File "/root/open_ce/build_feedstock.py", line 166, in build_feedstock_from_command
    conda_build.api.build(os.path.join(os.getcwd(), recipe['path']),
  File "/opt/conda/lib/python3.9/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/opt/conda/lib/python3.9/site-packages/conda_build/build.py", line 3068, in build_tree
    packages_from_this = build(metadata, stats,
  File "/opt/conda/lib/python3.9/site-packages/conda_build/build.py", line 2120, in build
    m.parse_until_resolved(allow_no_other_outputs=True)
  File "/opt/conda/lib/python3.9/site-packages/conda_build/metadata.py", line 1061, in parse_until_resolved
    self.parse_again(permit_undefined_jinja=True, allow_no_other_outputs=allow_no_other_outputs,
  File "/opt/conda/lib/python3.9/site-packages/conda_build/metadata.py", line 1001, in parse_again
    self.meta = parse(self._get_contents(permit_undefined_jinja,
  File "/opt/conda/lib/python3.9/site-packages/conda_build/metadata.py", line 1568, in _get_contents
    env.globals.update(environ.get_dict(m=self, skip_build_id=skip_build_id))
  File "/opt/conda/lib/python3.9/site-packages/conda_build/environ.py", line 282, in get_dict
    d.update(meta_vars(m, skip_build_id=skip_build_id))
  File "/opt/conda/lib/python3.9/site-packages/conda_build/environ.py", line 486, in meta_vars
    d.update(get_git_info(git_exe, git_dir, meta.config.debug))
  File "/opt/conda/lib/python3.9/site-packages/conda_build/environ.py", line 193, in get_git_info
    from conda._vendor.auxlib.packaging import _get_version_from_git_tag
ModuleNotFoundError: No module named 'conda._vendor.auxlib'
[OPEN-CE-ERROR]-11 Unable to build recipe: numactl-feedstock
[OPEN-CE-ERROR]-11 Unable to build recipe: numactl-feedstock

Remove open-ce-builder version from generated image names

When using the build image command, the open-ce-builder version is being added to the image name. Instead, we would like the --git_tag_for_env value that was used to create the conda environment file added to the image name. This value can be saved as a comment within the generated conda environment files.

Build failure on ubi9 images on x86

The build for packages is failing with following error on ubi9 container :

perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory

[BUG] output_folder should be indexed even if no package is built in it

Describe the bug
When no package is built but conda env files are generated using remote channels, the output_folder remains empty but is listed in the conda env file as a channel. So, even that empty output_folder should be conda indexed to be detected as a right channel, otherwise using conda env file gives error.

To Reproduce
Steps to reproduce the behavior, include the script and all arguments.

Expected behavior
A clear and concise description of what you expected to happen.

Execution Logs
If applicable, add logs to help explain your problem.

System (please complete the following information):

OS: [e.g. RHEL, Windows]
Architecture [e.g. Power, x86]

Additional context
Add any other context about the problem here.

[BUG] Build fails when using podman as root

Describe the bug
Using the --container_build argument with podman when running podman as root fails.

To Reproduce
Steps to reproduce the behavior, include the script and all arguments.

./open-ce-builder/open_ce/open-ce build env --container_build  --python_version 3.7 --build_type cuda --cuda_version 11.0  open-ce/envs/opence-env.yaml --conda_build_config open-ce/conda_build_config.yaml

Expected behavior
The build continues as normal.

Execution Logs

TEP 13: RUN export PATH="${PATH}" &&     echo "PATH="${PATH}"" >> ${HOME}/.profile &&     $CONDA_HOME/bin/conda config --system --add envs_dirs $CONDA_HOME/envs &&     $CONDA_HOME/bin/conda config --system --add pkgs_dirs $CONDA_HOME/pkgs &&     $CONDA_HOME/bin/conda config --system --set always_yes true &&     $CONDA_HOME/bin/conda config --system --set auto_update_conda false &&     $CONDA_HOME/bin/conda config --system --set notify_outdated_conda false &&     mkdir -p $CONDA_HOME/conda-bld &&     mkdir -p $HOME/.cache &&     $CONDA_HOME/bin/conda --version &&     sudo rm -f /run/nologin &&     yes | ssh-keygen -f $HOME/.ssh/id_rsa -q -t rsa -N '' && cp $HOME/.ssh/id_rsa.pub $HOME/.ssh/authorized_keys &&     echo ". $CONDA_HOME/etc/profile.d/conda.sh" >> ${HOME}/.bashrc &&     echo "export PYTHONPATH=${PYTHONPATH}:$HOME" >> ${HOME}/.bashrc &&     echo "conda activate base" >> ${HOME}/.bashrc
conda 4.10.1
STEP 14: COMMIT open-ce:open-ce-builder-cuda11.0
--> ac66f76631a
ac66f76631ab6d4d3b831ea671d241e39b6ac843b661459d98e3d186d06d535e
5b6299d58eac848f2431a16b5695d718e4f129d7a4f9f2c9ae083d67bbe2a210
open-ce-builder-20210416122040
Traceback (most recent call last):
  File "/home/builder/open_ce/open-ce", line 21, in <module>
    from open_ce import inputs
ModuleNotFoundError: No module named 'open_ce'
Error: exec session exited with non-zero exit code 1: OCI runtime error
5b6299d58eac848f2431a16b5695d718e4f129d7a4f9f2c9ae083d67bbe2a210
[OPEN-CE-ERROR-4] Error executing build in container: "open-ce-builder-20210416122040"

[BUG] open-ce-common-utils.sh script doesn't work with `--container_build` option

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior, include the script and all arguments.

Expected behavior
A clear and concise description of what you expected to happen.

Execution Logs
If applicable, add logs to help explain your problem.

System (please complete the following information):

OS: [e.g. RHEL, Windows]
Architecture [e.g. Power, x86]

Additional context
Add any other context about the problem here.

Builder Enhancements

Docker build images should have newer conda-build and conda versions
output_folder should be indexed even if no package is built in it
Description:
When no package is built but conda env files are generated using remote channels, the output_folder remains empty but is listed in the conda env file as a channel. So, even that empty output_folder should be conda indexed to be detected as a right channel, otherwise using conda env file gives error.
open-ce-common-utils.sh script doesn't work with --container_build option
Support --build_ffmpeg and --fips with --container_build option

Reduce size of built images

Images coming out of the open-ce build image command can be very large since they include copies of the generated channel. There is also a chown command on the channel which is doubling the size.

For this issue, determine if there is a way to create the image without including the channel at all in the final image. Newer versions of docker or podman might be needed to accomplish this.

pylint complaints observed while using pylint version compatible with python 3.10

Below complaints are seen when using pylint v2.12.2 which is needed to support python 3.10:

************* Module open_ce.build_env
open_ce/build_env.py:64:32: C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)
************* Module open_ce.build_image
open_ce/build_image.py:42:13: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/build_image.py:119:77: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.build_tree
open_ce/build_tree.py:55:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/build_tree.py:194:38: R1735: Consider using {} instead of dict() (use-dict-literal)
open_ce/build_tree.py:195:32: R1735: Consider using {} instead of dict() (use-dict-literal)
open_ce/build_tree.py:196:32: R1735: Consider using {} instead of dict() (use-dict-literal)
open_ce/build_tree.py:259:22: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/build_tree.py:427:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/build_tree.py:471:26: R1735: Consider using {} instead of dict() (use-dict-literal)
open_ce/build_tree.py:513:25: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.conda_env_file_generator
open_ce/conda_env_file_generator.py:70:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/conda_env_file_generator.py:83:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/conda_env_file_generator.py:98:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
************* Module open_ce.conda_utils
open_ce/conda_utils.py:126:18: R1734: Consider using [] instead of list() (use-list-literal)
open_ce/conda_utils.py:160:27: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/conda_utils.py:170:11: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.container_build
open_ce/container_build.py:208:10: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.env_config
open_ce/env_config.py:77:52: E1101: Instance of 'list' has no 'get' member (no-member)
open_ce/env_config.py:80:34: E1101: Instance of 'list' has no 'get' member (no-member)
open_ce/env_config.py:86:86: E1101: Instance of 'list' has no 'keys' member (no-member)
open_ce/env_config.py:88:12: E1126: Sequence index is not an int, slice, or instance with __index__ (invalid-sequence-index)
open_ce/env_config.py:90:47: E1101: Instance of 'list' has no 'keys' member (no-member)
open_ce/env_config.py:95:39: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.errors
open_ce/errors.py:108:18: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/errors.py:116:10: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module open_ce.get_licenses
open_ce/get_licenses.py:101:19: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/get_licenses.py:127:14: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/get_licenses.py:136:14: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/get_licenses.py:198:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:207:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:236:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:253:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:260:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:283:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:344:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:388:19: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
open_ce/get_licenses.py:426:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/get_licenses.py:457:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
************* Module open_ce.inputs
open_ce/inputs.py:60:48: R1734: Consider using [] instead of list() (use-list-literal)
open_ce/inputs.py:285:45: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/inputs.py:370:26: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/inputs.py:386:37: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/inputs.py:425:60: R1734: Consider using [] instead of list() (use-list-literal)
************* Module open_ce.test_feedstock
open_ce/test_feedstock.py:132:57: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/test_feedstock.py:223:47: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/test_feedstock.py:236:47: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/test_feedstock.py:265:23: R1735: Consider using {} instead of dict() (use-dict-literal)
open_ce/test_feedstock.py:283:23: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/test_feedstock.py:284:29: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/test_feedstock.py:286:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
************* Module open_ce.utils
open_ce/utils.py:57:60: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/utils.py:111:43: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/utils.py:118:47: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/utils.py:129:43: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/utils.py:135:14: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
open_ce/utils.py:277:28: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
open_ce/utils.py:279:28: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
open_ce/utils.py:293:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/utils.py:298:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/utils.py:366:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
open_ce/utils.py:392:11: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
************* Module open_ce.validate_config
open_ce/validate_config.py:61:25: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/validate_config.py:63:29: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
open_ce/validate_config.py:65:10: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)

------------------------------------------------------------------
Your code has been rated at 9.58/10 (previous run: 9.81/10, -0.23)

Add a CI test for using the `--container_build` option

We currently don't have any CI tests that use the --container_build option. Determine if there is a way to do this with GitHub actions. Possibly build a small recipe using the --container_build option.

[BUG] Crash in Validate Config with Bad Location

Describe the bug
When running validate config the builder crashes with the following message if the env file doesn't exist.

$ open-ce validate config --python_versions 3.7 --build_types cuda --repository_folder /tmp/bryant/test ./invalid-folder/invalid-env.yaml 
Traceback (most recent call last):
  File "/mnt/pai/home/bnelson/anaconda3/envs/dev/bin/open-ce", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/pai/home/bnelson/git/open-ce-builder/open_ce/open-ce", line 129, in <module>
    _main()
  File "/mnt/pai/home/bnelson/git/open-ce-builder/open_ce/open-ce", line 124, in _main
    args = inputs.parse_args(parser, arg_strings)
  File "/mnt/pai/home/bnelson/git/open-ce-builder/open_ce/inputs.py", line 307, in parse_args
    _create_env_config_paths(args)
  File "/mnt/pai/home/bnelson/git/open-ce-builder/open_ce/inputs.py", line 277, in _create_env_config_paths
    organization = os.path.basename(args.git_location)
AttributeError: 'Namespace' object has no attribute 'git_location'

To Reproduce

$ open-ce validate config --python_versions 3.7 --build_types cuda --repository_folder /tmp/bryant/test ./invalid-folder/invalid-env.yaml

Expected behavior
An error message should be shown stating that the env file doesn't exist.

Write scripts to tag bug fixes for old release of open-ce

The current scripts for tagging a version of Open-CE only work on the main branch of a feedstock. When tagging an older version of Open-CE, we might need to tag commits on release branches. The tagging script could be modified to:

Take an existing tag as an argument.
Tag the most recent commit of the most recent branch that includes that provided tag.

[REQUEST] verify that UBI8 builds are supported via --container_build option

Verify that --container_build option works for UBI8 builds.

[REQUEST] Add Schema Versioning

Describe the solution you'd like
We need a way to specify what version of tools an Open-CE file is compatible with and give a reasonable error about upgrading.

A solution we've discussed is having an builder-version field to Open-CE files, and having the tools check that field and displaying an error if the builder is outside of that version.

Something like:

builder_version: >=1.4.0,<1.5.0

packages:
  - feedstock : cdts
    runtime_package : False
  - feedstock : bazel
    runtime_package : False

Allow `open-ce get licenses` to find pip packages

The open-ce get licenses command can find license information for conda packages listed within a conda environment file, but it can't find information for pip packages listed within a conda environment file. There should be a way to find pip packages installed within the environment and determine where the packages are located. Once that is done, the license information from pip packages should be available.

[BUG] Auto-tagging script fails to tag feedstocks marked only for cpu build type

Describe the bug
Feedstocks mentioned in open-ce/envs/apache-beam.yaml or similar which lists packages only for build_type == cpu are not aut-tagged by tagging script create_opence_release.py. We need to tag them manually.

To Reproduce
Merge Release creation PR.

Expected behavior
All the feedstocks should be tagged for a particular opence release.

Execution Logs

System (please complete the following information):

OS: [e.g. RHEL, Windows] RHEL 7/8
Architecture [e.g. Power, x86]: Both

Use UBI9 images for Builder 12

Create UBI9 images for CUDA 11.8
Use UBI9 images for Builder 12.

[BUG] Remote feedstocks clone fails if git_tag is not present and git_tag_for_env is present

Describe the bug
In any env file, if a remote feedstock is specified with no git_tag and the env file has git_tag_for_env, then clone of the repo is done but checkout to git_tag_for_env fails. And this causes the build to stop or that feedstock gets missing.

To Reproduce
Following sample of env file can lead to the error -

packages:
   - feedstock: https://github.com/conda-forge/absl-py-feedstock
   - tensorboard
git_tag_for_env: open-ce-v1.7.0

Expected behavior
git_tag should be made mandatory in case it is not present for remote feedstocks. So, an error should be thrown if git_tag is not specified.

System (please complete the following information):

OS: [e.g. RHEL, Windows]: RHEL 7/8
Architecture [e.g. Power, x86]: Power, x86, s390x

[REQUEST] Build With System MPI

Describe the solution you'd like
Currently the builder provides the --mpi_type system variant flag to build with the system MPI.

The goal of that was primarily to allow users to build with a system installation of spectrum MPI.

However, on some systems there is an issue where the mpi compilers can't be found when build Horovod.

Describe alternatives you've considered
Passing through the the MPI_DIR env var used to work, but no longer does.
open-ce/horovod-feedstock#23

Now we are just disabling the conda compilers when using system MPI, but that's not ideal.
open-ce/horovod-feedstock#28

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Validation and Env File Gen not Limited to Variants

When validation is taking place and when the conda env files are being generated, the full tree is being used, which leads to non-variant packages ending up in the conda env file.

Set Py310 and CUDA 11.4 as default values for respective options

Update the default values for Python and CUDA to 3.10 and CUDA 11.4 respectively.

[BUG] - conda-bld/work - file not found

I am trying to compile the latest open-ce version and I am getting this error:

conda-bld/work - file not found

`HEAD is now at 8eb554c updated v1.5.8
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/shutil.py", line 791, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/home/florin/anaconda3/envs/opence/conda-bld/work' -> '/home/florin/anaconda3/envs/opence/conda-bld/fire_1651413312773/work'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/utils.py", line 384, in _run_helper
    return func(*args, **kwargs)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 327, in _create_commands_helper
    retval = _create_commands(repo_dir,
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 567, in _create_commands
    packages, version, run_deps, host_deps, build_deps, test_deps, output_files = _get_package_dependencies(
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 130, in _get_package_dependencies
    metas = conda_utils.render_yaml(path, variants, variant_config_files)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/conda_utils.py", line 53, in render_yaml
    metas = conda_build.api.render(path,
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/conda_build/api.py", line 42, in render
    metadata_tuples = render_recipe(recipe_path, bypass_env_check=bypass_env_check,
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/conda_build/render.py", line 838, in render_recipe
    m.config.compute_build_id(m.name(), m.version(), reset=reset_build_id)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/conda_build/config.py", line 611, in compute_build_id
    shutil.move(old_dir, work_dir)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/home/florin/anaconda3/envs/opence/conda-bld/work'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/florin/anaconda3/envs/opence/bin/open-ce", line 137, in <module>
    _main()
  File "/home/florin/anaconda3/envs/opence/bin/open-ce", line 133, in _main
    args.func(args)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_env.py", line 84, in build_env
    build_tree = construct_build_tree(args)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 681, in construct_build_tree
    return BuildTree(env_config_files=args.env_config_file,
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 202, in __init__
    variant_tree, external_deps, channels = self._create_nodes(variant)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/build_tree.py", line 314, in _create_nodes
    commands = utils.run_in_parallel(self._create_commands_helper, create_commands_args)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/site-packages/open_ce/utils.py", line 395, in run_in_parallel
    retval = pool.starmap(_run_helper, new_args)
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/florin/anaconda3/envs/opence/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/home/florin/anaconda3/envs/opence/conda-bld/work'`

Any advice?
Rgds,
FM

[BUG] LightGBM Feedstock Merge Workflow Failure

Describe the bug
https://github.com/open-ce/LightGBM-feedstock/runs/2241192667

[BUG] Pytest Coverage Thread Issues

Describe the bug

With the addition of the multi-threading to builder the following messages show up when running pytest:

overage.py warning: Data file '/mnt/pai/home/bnelson/git/open-ce-builder/.coverage.dlw05.156326.606137' doesn't seem to be a coverage data file: Couldn't use data file '/mnt/pai/home/bnelson/git/open-ce-builder/.coverage.dlw05.156326.606137': no such table: coverage_schema

Also, many .coverage* files are left over after running the tests.

We see these errors in our Github workflow runs also.

To Reproduce

The following will reproduce the messages:

pytest tests/get_graph_test.py

Additional context

This seems to be a known issue with PyTest and multi-threading in python, but it looks like there is a workaround.

open-ce / open-ce-builder Goto Github PK

open-ce-builder's Introduction

Open-CE Builder

GETTING STARTED

Requirements

CUDA Requirements

Installing the Open-CE Build Tools

Open-CE compatibility with Open-CE Builder

Building a Collection of Packages

Power10 MMA Optimization

Building Packages

Running Packages

FFMPEG with reduced codecs

Building within a container

Building a Single Feedstock

Installing Packages

Testing Packages

Creating Container Image with Open-CE Packages installed

Contributions

open-ce-builder's People

Contributors

Stargazers

Watchers

Forkers

open-ce-builder's Issues

Recommend Projects

Recommend Topics

Recommend Org