bioconda / bioconda-utils Goto Github PK

View Code? Open in Web Editor NEW

95.0 95.0 133.0 5.22 MB

Utilities for building and managing bioconda recipes

License: MIT License

Python 98.04% Shell 0.02% Dockerfile 0.37% HTML 0.78% CSS 0.79%

bioconda-utils's People

Contributors

Stargazers

Watchers

Forkers

johnnygenomics yixf-self jfallmann lparsons rvalieris jhcepas acaprez jerowe nsoranzo olgabot simonvh cbrueffer henningtimm epruesse jtratner frodepedersen mbargull sachalau yesimon eseiler mehrdad-shokri daler mathiashaudgaard jdblischak arnekr boll3 james0032 silceballos brevans luispedro kyleabeauchamp mrlmachado mikecormier dlaehnemann jvolkening nh13 dpryan79 nturaga ucf-arcc tfursten qwarctick cogentherapeutics druvus erictleung stain sndrtj hogfeldt 18874851654 windydreamer yanghui2048 enformatik juke34 gwaybio crazyyeast yusugihara dizak roryk a-neugodov georgie-ru tuvangezer matthiaskoenig blankenberg yujiajun96 gianmp94 nathanweeks leej3 jing-xinxing cshinn363 fatima967 marcelm grst dariober fruit-loops davebx pshahnazari david-parsons qian-elizabeth literallyuniquelogin nilshoffmann northstarpls mbhall88 bonomali sukram247 martenson zeehio balharbi nextstrain alshai siyangming ralatsdc fxwiegand antonioaug xiaofengdong83 x278645818 c-mertes dr-masum raivivek autumnfe79 jpfeuffer peterjc

bioconda-utils's Issues

cross-reference conda-forge and bioconda channels

Need a good mechanism for detecting whether a package on bioconda has been replaced by one on conda-forge and can therefore be moved to a recipes-deprecated subdir.

xref bioconda/bioconda-recipes#2796

CondaValueError: Could not find environment

Hello,

I have just done a fresh install of bioconda-utils in a new conda installation. When I use bioconda-utils build --docker on a Linux machine I get the errors below. If I remove the --docker option, it works.

Could you please let me know your thoughts?

Best regards,
Sebastian

bioconda-utils build recipes config.yml --docker --loglevel=info --packages cgat-scripts --loglevel=debug
14:58:17 BIOCONDA INFO DOCKER: Pulling docker image condaforge/linux-anvil
14:58:19 BIOCONDA DEBUG DOCKER: stdout+stderr:
Using default tag: latest
latest: Pulling from condaforge/linux-anvil
Digest: sha256:dd53ef1792f777fa8b2abf30221bf9beb6cb6a81d158d498f0219c4d475696cd
Status: Image is up to date for condaforge/linux-anvil:latest

14:58:19 BIOCONDA INFO DOCKER: Done pulling image
14:58:19 BIOCONDA INFO DOCKER: Building image "tmp-bioconda-builder" from /tmp/tmpwfwm7ysb
14:58:19 BIOCONDA DEBUG Dockerfile:

FROM condaforge/linux-anvil
COPY requirements.txt /tmp/requirements.txt

RUN /opt/conda/bin/conda config --add channels defaults
RUN /opt/conda/bin/conda config --add channels conda-forge
RUN /opt/conda/bin/conda config --add channels bioconda
RUN /opt/conda/bin/conda install --file /tmp/requirements.txt

....
15:02:19 BIOCONDA INFO DOCKER: Built docker image tag=tmp-bioconda-builder
15:02:19 BIOCONDA INFO blacklist: blast/2.2.21, cap-mirseq, denovogear, detonate, mgkit, mothur/1.36.1, perl-font-afm, r-histonehmm
15:02:19 BIOCONDA DEBUG get_recipes(recipes, package='['cgat-scripts']'): cgat-scripts
15:02:19 BIOCONDA DEBUG recipes/cgat-scripts
15:02:19 BIOCONDA DEBUG recipes: ['recipes/cgat-scripts']
15:02:19 BIOCONDA INFO Filtering recipes
15:02:21 BIOCONDA DEBUG recipes: ['recipes/cgat-scripts']

15:02:22 BIOCONDA DEBUG FILTER: building cgat-scripts-0.2.7-np112py27r3.3.2_0.tar.bz2 because it is not in channels and does not define skip
15:02:22 BIOCONDA DEBUG FILTER: building cgat-scripts-0.2.7-np112py35r3.3.2_0.tar.bz2 because it is not in channels and does not define skip
15:02:22 BIOCONDA DEBUG FILTER: building cgat-scripts-0.2.7-np112py36r3.3.2_0.tar.bz2 because it is not in channels and does not define skip
Filtering 1 of 1 (100.0%) recipes/cgat-scripts
recipes/cgat-scripts
15:02:22 BIOCONDA INFO Building and testing 1 recipes in total
15:02:22 BIOCONDA INFO Recipes to build: 
cgat-scripts
15:02:22 BIOCONDA INFO Building and testing subdag 1 of 1 (1 recipes)
15:02:22 BIOCONDA INFO BUILD START recipes/cgat-scripts, env: CONDA_ALLOW_SOFTLINKS=false;CONDA_BOOST=1.61;CONDA_DEFAULT_ENV=root;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin;CONDA_PERL=5.22.0;CONDA_PREFIX=/conda/sebastian/py3;CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\];CONDA_PY=27;CONDA_R=3.3.2;CONDA_ROOT=/conda/sebastian/py3;LANG=C;MACOSX_DEPLOYMENT_TARGET=10.9;PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin
15:02:22 BIOCONDA DEBUG build_args: ['--no-anaconda-upload']
15:02:22 BIOCONDA DEBUG channel_args: ['--channel', 'bioconda', '--channel', 'conda-forge', '--channel', 'defaults']
15:02:23 BIOCONDA DEBUG DOCKER: Container build script: 

#!/bin/bash
set -e

conda install conda-build=2.1.16 conda=4.3.21  > /dev/null 2>&1

# Add the host's mounted conda-bld dir so that we can use its contents as
# dependencies for building this recipe.
#
# Note that if the directory didn't exist on the host, then the staging area
# will exist in the container but will be empty.  Channels expect at least
# a linux-64 and noarch directory within that directory, so we make sure it
# exists before adding the channel.
mkdir -p /opt/host-conda-bld/linux-64
mkdir -p /opt/host-conda-bld/noarch
conda config --add channels file:///opt/host-conda-bld  > /dev/null 2>&1

# The actual building...
# we explicitly point to the meta.yaml, in order to keep
# conda-build from building all subdirectories
conda build --channel bioconda --channel conda-forge --channel defaults --no-anaconda-upload /opt/recipe/meta.yaml 2>&1

# Identify the output package
OUTPUT_DIR=$(dirname $(conda build /opt/recipe/meta.yaml --output 2> /dev/null))
OUTPUT=$OUTPUT_DIR/cgat-scripts-0.2.7-np112py27r3.3.2_0.tar.bz2


# Some args to conda-build make it run and exit 0 without creating a package
# (e.g., -h or --skip-existing), so check to see if there's anything to copy
# over first.
if [[ -e $OUTPUT ]]; then

    # Copy over the recipe from where the container built it to the mounted
    # conda-bld dir from the host. The arch will be either linux-64 or noarch.
    cp $OUTPUT /opt/host-conda-bld/linux-64

    # Ensure permissions are correct on the host.
    HOST_USER=1362
    chown $HOST_USER:$HOST_USER /opt/host-conda-bld/linux-64/$(basename $OUTPUT)

    conda index /opt/host-conda-bld/linux-64 > /dev/null 2>&1
fi

15:02:23 BIOCONDA DEBUG DOCKER: cmd: ['docker', 'run', '--net', 'host', '--rm', '-v', '/tmp/tmpxn2pcntf/build_script.bash:/opt/build_script.bash', '-v', '/conda/sebastian/py3/conda-bld:/opt/host-conda-bld', '-v', '/conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe', '-e', 'CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_PREFIX=/conda/sebastian/py3', '-e', 'CONDA_GSL=1.16', '-e', 'CONDA_DEFAULT_ENV=root', '-e', 'CONDA_GMP=5.1', '-e', 'PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_ROOT=/conda/sebastian/py3', '-e', 'MACOSX_DEPLOYMENT_TARGET=10.9', '-e', 'CONDA_PY=27', '-e', 'CONDA_ALLOW_SOFTLINKS=false', '-e', 'CONDA_BOOST=1.61', '-e', 'LANG=C', '-e', 'CONDA_HDF5=1.8.17', '-e', 'CONDA_NPY=112', '-e', 'CONDA_HTSLIB=1.5', '-e', 'CONDA_PERL=5.22.0', '-e', 'CONDA_NCURSES=5.9', '-e', 'CONDA_R=3.3.2', '-e', 'CONDA_PS1_BACKUP=\\n\\[\x1b[1m\\]\\[\x1b[32m\\][\\d \\t] \\u@\\h:\\w\\n\\$ \\[\x1b(B\x1b[m\\]', 'tmp-bioconda-builder', '/bin/bash', '/opt/build_script.bash']
.15:02:39 BIOCONDA ERROR COMMAND FAILED: docker run --net host --rm -v /tmp/tmpxn2pcntf/build_script.bash:/opt/build_script.bash -v /conda/sebastian/py3/conda-bld:/opt/host-conda-bld -v /conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe -e CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_PREFIX=/conda/sebastian/py3 -e CONDA_GSL=1.16 -e CONDA_DEFAULT_ENV=root -e CONDA_GMP=5.1 -e PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_ROOT=/conda/sebastian/py3 -e MACOSX_DEPLOYMENT_TARGET=10.9 -e CONDA_PY=27 -e CONDA_ALLOW_SOFTLINKS=false -e CONDA_BOOST=1.61 -e LANG=C -e CONDA_HDF5=1.8.17 -e CONDA_NPY=112 -e CONDA_HTSLIB=1.5 -e CONDA_PERL=5.22.0 -e CONDA_NCURSES=5.9 -e CONDA_R=3.3.2 -e CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\] tmp-bioconda-builder /bin/bash /opt/build_script.bash
15:02:39 BIOCONDA ERROR STDOUT+STDERR:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Can't read /etc/scl/prefixes/devtoolset-2, devtoolset-2 is probably not installed.
bash: uname: command not found
bash: uname: command not found

CondaValueError: Could not find environment: /conda/sebastian/py3

bash: sed: command not found
bash: grep: command not found
bash: grep: command not found
/opt/build_script.bash: line 14: mkdir: command not found


15:03:23 BIOCONDA ERROR BUILD FAILED recipes/cgat-scripts, CONDA_BOOST=1.61;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PERL=5.22.0;CONDA_PY=27;CONDA_R=3.3.2;MACOSX_DEPLOYMENT_TARGET=10.9
15:03:24 BIOCONDA INFO BUILD START recipes/cgat-scripts, env: CONDA_ALLOW_SOFTLINKS=false;CONDA_BOOST=1.61;CONDA_DEFAULT_ENV=root;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin;CONDA_PERL=5.22.0;CONDA_PREFIX=/conda/sebastian/py3;CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\];CONDA_PY=35;CONDA_R=3.3.2;CONDA_ROOT=/conda/sebastian/py3;LANG=C;MACOSX_DEPLOYMENT_TARGET=10.9;PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin
15:03:24 BIOCONDA DEBUG build_args: ['--no-anaconda-upload']
15:03:24 BIOCONDA DEBUG channel_args: ['--channel', 'bioconda', '--channel', 'conda-forge', '--channel', 'defaults']
15:03:24 BIOCONDA DEBUG DOCKER: Container build script: 

#!/bin/bash
set -e

conda install conda-build=2.1.16 conda=4.3.21  > /dev/null 2>&1

# Add the host's mounted conda-bld dir so that we can use its contents as
# dependencies for building this recipe.
#
# Note that if the directory didn't exist on the host, then the staging area
# will exist in the container but will be empty.  Channels expect at least
# a linux-64 and noarch directory within that directory, so we make sure it
# exists before adding the channel.
mkdir -p /opt/host-conda-bld/linux-64
mkdir -p /opt/host-conda-bld/noarch
conda config --add channels file:///opt/host-conda-bld  > /dev/null 2>&1

# The actual building...
# we explicitly point to the meta.yaml, in order to keep
# conda-build from building all subdirectories
conda build --channel bioconda --channel conda-forge --channel defaults --no-anaconda-upload /opt/recipe/meta.yaml 2>&1

# Identify the output package
OUTPUT_DIR=$(dirname $(conda build /opt/recipe/meta.yaml --output 2> /dev/null))
OUTPUT=$OUTPUT_DIR/cgat-scripts-0.2.7-np112py35r3.3.2_0.tar.bz2


# Some args to conda-build make it run and exit 0 without creating a package
# (e.g., -h or --skip-existing), so check to see if there's anything to copy
# over first.
if [[ -e $OUTPUT ]]; then

    # Copy over the recipe from where the container built it to the mounted
    # conda-bld dir from the host. The arch will be either linux-64 or noarch.
    cp $OUTPUT /opt/host-conda-bld/linux-64

    # Ensure permissions are correct on the host.
    HOST_USER=1362
    chown $HOST_USER:$HOST_USER /opt/host-conda-bld/linux-64/$(basename $OUTPUT)

    conda index /opt/host-conda-bld/linux-64 > /dev/null 2>&1
fi

15:03:24 BIOCONDA DEBUG DOCKER: cmd: ['docker', 'run', '--net', 'host', '--rm', '-v', '/tmp/tmpfzn1vnjs/build_script.bash:/opt/build_script.bash', '-v', '/conda/sebastian/py3/conda-bld:/opt/host-conda-bld', '-v', '/conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe', '-e', 'CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_PREFIX=/conda/sebastian/py3', '-e', 'CONDA_GSL=1.16', '-e', 'CONDA_DEFAULT_ENV=root', '-e', 'CONDA_GMP=5.1', '-e', 'PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_ROOT=/conda/sebastian/py3', '-e', 'MACOSX_DEPLOYMENT_TARGET=10.9', '-e', 'CONDA_PY=35', '-e', 'CONDA_ALLOW_SOFTLINKS=false', '-e', 'CONDA_BOOST=1.61', '-e', 'LANG=C', '-e', 'CONDA_HDF5=1.8.17', '-e', 'CONDA_NPY=112', '-e', 'CONDA_HTSLIB=1.5', '-e', 'CONDA_PERL=5.22.0', '-e', 'CONDA_NCURSES=5.9', '-e', 'CONDA_R=3.3.2', '-e', 'CONDA_PS1_BACKUP=\\n\\[\x1b[1m\\]\\[\x1b[32m\\][\\d \\t] \\u@\\h:\\w\\n\\$ \\[\x1b(B\x1b[m\\]', 'tmp-bioconda-builder', '/bin/bash', '/opt/build_script.bash']
.15:03:40 BIOCONDA ERROR COMMAND FAILED: docker run --net host --rm -v /tmp/tmpfzn1vnjs/build_script.bash:/opt/build_script.bash -v /conda/sebastian/py3/conda-bld:/opt/host-conda-bld -v /conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe -e CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_PREFIX=/conda/sebastian/py3 -e CONDA_GSL=1.16 -e CONDA_DEFAULT_ENV=root -e CONDA_GMP=5.1 -e PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_ROOT=/conda/sebastian/py3 -e MACOSX_DEPLOYMENT_TARGET=10.9 -e CONDA_PY=35 -e CONDA_ALLOW_SOFTLINKS=false -e CONDA_BOOST=1.61 -e LANG=C -e CONDA_HDF5=1.8.17 -e CONDA_NPY=112 -e CONDA_HTSLIB=1.5 -e CONDA_PERL=5.22.0 -e CONDA_NCURSES=5.9 -e CONDA_R=3.3.2 -e CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\] tmp-bioconda-builder /bin/bash /opt/build_script.bash
15:03:40 BIOCONDA ERROR STDOUT+STDERR:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Can't read /etc/scl/prefixes/devtoolset-2, devtoolset-2 is probably not installed.
bash: uname: command not found
bash: uname: command not found

CondaValueError: Could not find environment: /conda/sebastian/py3

bash: sed: command not found
bash: grep: command not found
bash: grep: command not found
/opt/build_script.bash: line 14: mkdir: command not found


15:04:25 BIOCONDA ERROR BUILD FAILED recipes/cgat-scripts, CONDA_BOOST=1.61;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PERL=5.22.0;CONDA_PY=35;CONDA_R=3.3.2;MACOSX_DEPLOYMENT_TARGET=10.9
15:04:26 BIOCONDA INFO BUILD START recipes/cgat-scripts, env: CONDA_ALLOW_SOFTLINKS=false;CONDA_BOOST=1.61;CONDA_DEFAULT_ENV=root;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin;CONDA_PERL=5.22.0;CONDA_PREFIX=/conda/sebastian/py3;CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\];CONDA_PY=36;CONDA_R=3.3.2;CONDA_ROOT=/conda/sebastian/py3;LANG=C;MACOSX_DEPLOYMENT_TARGET=10.9;PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin
15:04:26 BIOCONDA DEBUG build_args: ['--no-anaconda-upload']
15:04:26 BIOCONDA DEBUG channel_args: ['--channel', 'bioconda', '--channel', 'conda-forge', '--channel', 'defaults']
15:04:26 BIOCONDA DEBUG DOCKER: Container build script: 

#!/bin/bash
set -e

conda install conda-build=2.1.16 conda=4.3.21  > /dev/null 2>&1

# Add the host's mounted conda-bld dir so that we can use its contents as
# dependencies for building this recipe.
#
# Note that if the directory didn't exist on the host, then the staging area
# will exist in the container but will be empty.  Channels expect at least
# a linux-64 and noarch directory within that directory, so we make sure it
# exists before adding the channel.
mkdir -p /opt/host-conda-bld/linux-64
mkdir -p /opt/host-conda-bld/noarch
conda config --add channels file:///opt/host-conda-bld  > /dev/null 2>&1

# The actual building...
# we explicitly point to the meta.yaml, in order to keep
# conda-build from building all subdirectories
conda build --channel bioconda --channel conda-forge --channel defaults --no-anaconda-upload /opt/recipe/meta.yaml 2>&1

# Identify the output package
OUTPUT_DIR=$(dirname $(conda build /opt/recipe/meta.yaml --output 2> /dev/null))
OUTPUT=$OUTPUT_DIR/cgat-scripts-0.2.7-np112py36r3.3.2_0.tar.bz2


# Some args to conda-build make it run and exit 0 without creating a package
# (e.g., -h or --skip-existing), so check to see if there's anything to copy
# over first.
if [[ -e $OUTPUT ]]; then

    # Copy over the recipe from where the container built it to the mounted
    # conda-bld dir from the host. The arch will be either linux-64 or noarch.
    cp $OUTPUT /opt/host-conda-bld/linux-64

    # Ensure permissions are correct on the host.
    HOST_USER=1362
    chown $HOST_USER:$HOST_USER /opt/host-conda-bld/linux-64/$(basename $OUTPUT)

    conda index /opt/host-conda-bld/linux-64 > /dev/null 2>&1
fi

15:04:26 BIOCONDA DEBUG DOCKER: cmd: ['docker', 'run', '--net', 'host', '--rm', '-v', '/tmp/tmpko472l2m/build_script.bash:/opt/build_script.bash', '-v', '/conda/sebastian/py3/conda-bld:/opt/host-conda-bld', '-v', '/conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe', '-e', 'CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_PREFIX=/conda/sebastian/py3', '-e', 'CONDA_GSL=1.16', '-e', 'CONDA_DEFAULT_ENV=root', '-e', 'CONDA_GMP=5.1', '-e', 'PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin', '-e', 'CONDA_ROOT=/conda/sebastian/py3', '-e', 'MACOSX_DEPLOYMENT_TARGET=10.9', '-e', 'CONDA_PY=36', '-e', 'CONDA_ALLOW_SOFTLINKS=false', '-e', 'CONDA_BOOST=1.61', '-e', 'LANG=C', '-e', 'CONDA_HDF5=1.8.17', '-e', 'CONDA_NPY=112', '-e', 'CONDA_HTSLIB=1.5', '-e', 'CONDA_PERL=5.22.0', '-e', 'CONDA_NCURSES=5.9', '-e', 'CONDA_R=3.3.2', '-e', 'CONDA_PS1_BACKUP=\\n\\[\x1b[1m\\]\\[\x1b[32m\\][\\d \\t] \\u@\\h:\\w\\n\\$ \\[\x1b(B\x1b[m\\]', 'tmp-bioconda-builder', '/bin/bash', '/opt/build_script.bash']
.15:04:43 BIOCONDA ERROR COMMAND FAILED: docker run --net host --rm -v /tmp/tmpko472l2m/build_script.bash:/opt/build_script.bash -v /conda/sebastian/py3/conda-bld:/opt/host-conda-bld -v /conda/sebastian/bioconda-recipes/recipes/cgat-scripts:/opt/recipe -e CONDA_PATH_BACKUP=/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_PREFIX=/conda/sebastian/py3 -e CONDA_GSL=1.16 -e CONDA_DEFAULT_ENV=root -e CONDA_GMP=5.1 -e PATH=/conda/sebastian/py3/bin:/usr/lib64/qt-3.3/bin:/ifs/apps/system/git-2.13.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/ifs/home/sebastian/bin -e CONDA_ROOT=/conda/sebastian/py3 -e MACOSX_DEPLOYMENT_TARGET=10.9 -e CONDA_PY=36 -e CONDA_ALLOW_SOFTLINKS=false -e CONDA_BOOST=1.61 -e LANG=C -e CONDA_HDF5=1.8.17 -e CONDA_NPY=112 -e CONDA_HTSLIB=1.5 -e CONDA_PERL=5.22.0 -e CONDA_NCURSES=5.9 -e CONDA_R=3.3.2 -e CONDA_PS1_BACKUP=\n\[\]\[\][\d \t] \u@\h:\w\n\$ \[\] tmp-bioconda-builder /bin/bash /opt/build_script.bash
15:04:43 BIOCONDA ERROR STDOUT+STDERR:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Can't read /etc/scl/prefixes/devtoolset-2, devtoolset-2 is probably not installed.
bash: uname: command not found
bash: uname: command not found

CondaValueError: Could not find environment: /conda/sebastian/py3

bash: sed: command not found
bash: grep: command not found
bash: grep: command not found
/opt/build_script.bash: line 14: mkdir: command not found


15:05:26 BIOCONDA ERROR BUILD FAILED recipes/cgat-scripts, CONDA_BOOST=1.61;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_HDF5=1.8.17;CONDA_HTSLIB=1.5;CONDA_NCURSES=5.9;CONDA_NPY=112;CONDA_PERL=5.22.0;CONDA_PY=36;CONDA_R=3.3.2;MACOSX_DEPLOYMENT_TARGET=10.9
15:05:28 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
15:05:28 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe cgat-scripts-0.2.7-np112py27r3.3.2_0.tar.bz2, environment CONDA_GSL=1.16;CONDA_GMP=5.1;MACOSX_DEPLOYMENT_TARGET=10.9;CONDA_PY=27;CONDA_BOOST=1.61;CONDA_HDF5=1.8.17;CONDA_NPY=112;CONDA_HTSLIB=1.5;CONDA_PERL=5.22.0;CONDA_R=3.3.2;CONDA_NCURSES=5.9
15:05:28 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe cgat-scripts-0.2.7-np112py35r3.3.2_0.tar.bz2, environment CONDA_GSL=1.16;CONDA_GMP=5.1;MACOSX_DEPLOYMENT_TARGET=10.9;CONDA_PY=35;CONDA_BOOST=1.61;CONDA_HDF5=1.8.17;CONDA_NPY=112;CONDA_HTSLIB=1.5;CONDA_PERL=5.22.0;CONDA_R=3.3.2;CONDA_NCURSES=5.9
15:05:28 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe cgat-scripts-0.2.7-np112py36r3.3.2_0.tar.bz2, environment CONDA_GSL=1.16;CONDA_GMP=5.1;MACOSX_DEPLOYMENT_TARGET=10.9;CONDA_PY=36;CONDA_BOOST=1.61;CONDA_HDF5=1.8.17;CONDA_NPY=112;CONDA_HTSLIB=1.5;CONDA_PERL=5.22.0;CONDA_R=3.3.2;CONDA_NCURSES=5.9

`build recipes config.yml --docker` fails on OSX

The docker build fails with:
/opt/build_script.bash: /opt/build_script.bash: is a directory

This file is mounted into docker via -v /private/var/folders/XX/XXX...XXX/X/tmpXXX/build_script.bash:/opt/build_script.bash, where the path up to tmpXXX is the OSX tmpdir. This path is not mounted into the docker-machine VM running Linux, so it is not available in docker to mount. The result is (for some weird reason), that the mounted bash script appears as a directory.

Suggested fix: create the temporary folder for the bash script in ~/.local/tmp when operating on OSX.

AttributeError: 'dict' object has no attribute 'split'

Hi,

I get an AttributeError exception when doing:

bioconda-utils build recipes config.yml --mulled-test --packages cgat-scripts

Full output is here

I think the error was caused after adding a pip dependency on my meta.yaml file, like

requirements:
   build:
      - python 3.5
      - pip
         - bx-python==0.7.3

Do you know why?

Best regards,
Sebastian

Tests are skipped for pushes to the main bioconda-recipes repo.

Hello,

After re-reading the docs here:
http://bioconda.github.io/contribute-a-recipe.html#update-repo

I thought one can now push directly to bioconda-recipes without a fork. At least, that's what I understood from the following bit:

If you’re using a clone:

git checkout master
git pull origin master

However, after doing that I get: Tests are skipped for pushes to the main bioconda-recipes repo.

Could you please clarify this?

Many thanks,
Sebastian

errors when running simulate-travis.py

Hello,

I am using ./simulate-travis.py --packages mypackagename --loglevel=debug and getting the error below.

However, I don't know how to go from there. Do you have an idea?

First error:

ERROR bioconda_utils.build:build(112): BIOCONDA BUILD FAILED recipes/cgat-scripts, CONDA_BOOST=1.60;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_NCURSES=5.9;CONDA_NPY=110;CONDA_PERL=5.22.0;CONDA_PY=27;CONDA_R=3.3.1
ERROR bioconda_utils.build:build(113): COMMAND: docker run --net host --rm -v /tmp/tmpcpkgziem/build_script.bash:/opt/build_script.bash -v /sebastian/conda/conda3-build/bioconda/conda-bld:/opt/host-conda-bld -v /ifs/devel/system/conda/bioconda-recipes/recipes/cgat-scripts:/opt/recipe -e CONDA_PY=27 -e CONDA_GMP=5.1 -e CONDA_BOOST=1.60 -e CONDA_GSL=1.16 -e CONDA_NCURSES=5.9 -e CONDA_PERL=5.22.0 -e CONDA_R=3.3.1 -e CONDA_NPY=110 tmp-bioconda-builder /bin/bash /opt/build_script.bash
ERROR bioconda_utils.build:build(114): STDOUT+STDERR: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

seems to be explained here:
http://serverfault.com/questions/411307/cannot-set-terminal-process-group-during-su-to-another-user-as-login-shell

Second error:

ERROR bioconda_utils.build:build_recipes(355): BIOCONDA BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
ERROR bioconda_utils.build:build_recipes(366): BIOCONDA BUILD SUMMARY: FAILED recipe cgat-scripts-0.2.6-py27r3.3.1_0.tar.bz2, environment CONDA_PY=27;CONDA_PERL=5.22.0;CONDA_R=3.3.1;CONDA_NPY=110;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_BOOST=1.60;CONDA_NCURSES=5.9
Traceback (most recent call last):
  File "./simulate-travis.py", line 150, in <module>
    sp.run(['scripts/travis-run.sh'], env=env, universal_newlines=True, check=True)
  File "/sebastian/conda/conda3-build/bioconda/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['scripts/travis-run.sh']' returned non-zero exit status 1

does not explain why the build is failing for my recipe. How can I get it to print it?

Please see full error below.

Best regards,
Sebastian

./simulate-travis.py --packages cgat-scripts --loglevel=debug
+ bioconda-utils build recipes config.yml --docker --loglevel=info --mulled-test --packages cgat-scripts --loglevel=debug
INFO bioconda_utils.docker_utils:_pull_image(308): BIOCONDA DOCKER: Pulling docker image condaforge/linux-anvil
DEBUG bioconda_utils.docker_utils:_pull_image(310): BIOCONDA DOCKER: stdout+stderr: Using default tag: latest
latest: Pulling from condaforge/linux-anvil
Digest: sha256:a2e1bf0fae91b9f7a0305aa12efd7a0a3e80af2e13a7e8eba8363ff8425148aa
Status: Image is up to date for condaforge/linux-anvil:latest

INFO bioconda_utils.docker_utils:_pull_image(314): BIOCONDA DOCKER: Done pulling image
DEBUG bioconda_utils.docker_utils:_build_image(325): BIOCONDA DOCKER: Building image "tmp-bioconda-builder" from /tmp/tmpbckuak52
DEBUG bioconda_utils.docker_utils:_build_image(338): Dockerfile:

FROM condaforge/linux-anvil
COPY requirements.txt /tmp/requirements.txt
RUN /opt/conda/bin/conda install --file /tmp/requirements.txt

INFO bioconda_utils.docker_utils:_build_image(354): BIOCONDA DOCKER: Built docker image tag=tmp-bioconda-builder
INFO bioconda_utils.build:build_recipes(213): blacklist: bioconductor-bubbletree, bioconductor-cexor, bioconductor-csaw, bioconductor-dada2, bioconductor-deseq2/1.10.1, bioconductor-dexseq, bioconductor-diffbind, bioconductor-ebseq, bioconductor-genelendatabase/1.6.0, bioconductor-iranges/2.4.6, bioconductor-iranges/2.4.7, bioconductor-limma/3.26.7, bioconductor-limma/3.28.2, bioconductor-limma/3.28.6, bioconductor-minfi, bioconductor-mmdiff, bioconductor-qdnaseq, bioconductor-s4vectors/0.8.7, bioconductor-shortread, bioconductor-summarizedexperiment/1.0.2, bioconductor-systempiper, cap-mirseq, lefse, mgkit, mysqlclient, poretools/0.5.0, poretools/0.5.1, r-batchjobs, r-gsalib, r-hdrcde, r-knitr, r-ks, r-mixomics, r-mutoss, r-phonr, r-rainbow, r-readr, r-sartools/1.2.0, r-spp, rsem, triform2, vsearch
DEBUG bioconda_utils.utils:get_recipes(245): get_recipes(recipes, package='['cgat-scripts']'): cgat-scripts
DEBUG bioconda_utils.build:build_recipes(224): recipes/cgat-scripts
DEBUG bioconda_utils.build:build_recipes(229): recipes: ['recipes/cgat-scripts']
INFO bioconda_utils.build:build_recipes(248): Filtering recipes
DEBUG bioconda_utils.utils:filter_recipes(494): recipes: ['recipes/cgat-scripts']

DEBUG bioconda_utils.utils:tobuild(491): BIOCONDA FILTER: building cgat-scripts-0.2.6-py27r3.3.1_0.tar.bz2 because it is not in channels does not define skip, and force is not specified
DEBUG bioconda_utils.utils:tobuild(486): BIOCONDA FILTER: not building cgat-scripts-0.2.6-py34r3.3.1_0.tar.bz2 because it defines skip for this env
DEBUG bioconda_utils.utils:tobuild(486): BIOCONDA FILTER: not building cgat-scripts-0.2.6-py35r3.3.1_0.tar.bz2 because it defines skip for this env

INFO bioconda_utils.build:build_recipes(265): Building and testing 1 recipes in total
INFO bioconda_utils.build:build_recipes(266): Recipes to build: 
cgat-scripts
INFO bioconda_utils.build:build_recipes(303): Building and testing subdag 0 of 1 (1 recipes)
INFO bioconda_utils.build:build(58): BIOCONDA BUILD START recipes/cgat-scripts, env: CONDA_BOOST=1.60;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_NCURSES=5.9;CONDA_NPY=110;CONDA_PERL=5.22.0;CONDA_PY=27;CONDA_R=3.3.1
DEBUG bioconda_utils.build:build(74): build_args: ['--no-build-id', '--no-anaconda-upload']
DEBUG bioconda_utils.build:build(75): channel_args: ['--channel', 'bioconda', '--channel', 'r', '--channel', 'defaults', '--channel', 'conda-forge']
DEBUG bioconda_utils.docker_utils:build_recipe(391): BIOCONDA DOCKER: Container build script: 

#!/bin/bash
set -e

conda install conda-build=2.0.7

# Add the host's mounted conda-bld dir so that we can use its contents as
# dependencies for building this recipe.
#
# Note that if the directory didn't exist on the host, then the staging area
# will exist in the container but will be empty.  Channels expect at least
# a linux-64 directory within that directory, so we make sure it exists before
# adding the channel.
mkdir -p /opt/host-conda-bld/linux-64
conda config --add channels file:///opt/host-conda-bld

# The actual building....
conda build --channel bioconda --channel r --channel defaults --channel conda-forge --no-build-id --no-anaconda-upload /opt/recipe

# Identify the output package
OUTPUT=$(conda build /opt/recipe --output)

# Some args to conda-build make it run and exit 0 without creating a package
# (e.g., -h or --skip-existing), so check to see if there's anything to copy
# over first.
if [[ -e $OUTPUT ]]; then

    # Copy over the recipe from where the container built it to the mounted
    # conda-bld dir from the host. Since docker containers are Linux, we assume
    # here that we want the linux-64 arch.
    cp $OUTPUT /opt/host-conda-bld/linux-64

    # Ensure permissions are correct on the host.
    HOST_USER=1362
    chown $HOST_USER:$HOST_USER /opt/host-conda-bld/linux-64/$(basename $OUTPUT)

    conda index /opt/host-conda-bld/linux-64
fi

DEBUG bioconda_utils.docker_utils:build_recipe(413): BIOCONDA DOCKER: cmd: ['docker', 'run', '--net', 'host', '--rm', '-v', '/tmp/tmpcpkgziem/build_script.bash:/opt/build_script.bash', '-v', '/sebastian/conda/conda3-build/bioconda/conda-bld:/opt/host-conda-bld', '-v', '/ifs/devel/system/conda/bioconda-recipes/recipes/cgat-scripts:/opt/recipe', '-e', 'CONDA_PY=27', '-e', 'CONDA_GMP=5.1', '-e', 'CONDA_BOOST=1.60', '-e', 'CONDA_GSL=1.16', '-e', 'CONDA_NCURSES=5.9', '-e', 'CONDA_PERL=5.22.0', '-e', 'CONDA_R=3.3.1', '-e', 'CONDA_NPY=110', 'tmp-bioconda-builder', '/bin/bash', '/opt/build_script.bash']
ERROR bioconda_utils.build:build(112): BIOCONDA BUILD FAILED recipes/cgat-scripts, CONDA_BOOST=1.60;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_NCURSES=5.9;CONDA_NPY=110;CONDA_PERL=5.22.0;CONDA_PY=27;CONDA_R=3.3.1
ERROR bioconda_utils.build:build(113): COMMAND: docker run --net host --rm -v /tmp/tmpcpkgziem/build_script.bash:/opt/build_script.bash -v /sebastian/conda/conda3-build/bioconda/conda-bld:/opt/host-conda-bld -v /ifs/devel/system/conda/bioconda-recipes/recipes/cgat-scripts:/opt/recipe -e CONDA_PY=27 -e CONDA_GMP=5.1 -e CONDA_BOOST=1.60 -e CONDA_GSL=1.16 -e CONDA_NCURSES=5.9 -e CONDA_PERL=5.22.0 -e CONDA_R=3.3.1 -e CONDA_NPY=110 tmp-bioconda-builder /bin/bash /opt/build_script.bash
ERROR bioconda_utils.build:build(114): STDOUT+STDERR: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /opt/conda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-build-2.0.7          |           py35_0         275 KB  defaults

The following packages will be UPDATED:

    conda:       4.1.11-py35_1 conda-forge --> 4.1.12-py35_0 conda-forge
    conda-build: 2.0.4-py35_0  defaults    --> 2.0.7-py35_0  defaults   

Proceed ([y]/n)? 
Using Anaconda Cloud api site https://api.anaconda.org
Fetching packages ...
conda-build-2. 100% |###############################| Time: 0:00:00  12.78 MB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[      COMPLETE      ]|##################################################| 100%
Using Anaconda Cloud api site https://api.anaconda.org
BUILD START: cgat-scripts-0.2.6-py27r3.3.1_0
updating index in: /opt/conda/conda-bld/linux-64
updating index in: /opt/conda/conda-bld/noarch
Could not find URL: file:///opt/host-conda-bld/linux-64/

ERROR bioconda_utils.build:build_recipes(355): BIOCONDA BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
ERROR bioconda_utils.build:build_recipes(366): BIOCONDA BUILD SUMMARY: FAILED recipe cgat-scripts-0.2.6-py27r3.3.1_0.tar.bz2, environment CONDA_PY=27;CONDA_PERL=5.22.0;CONDA_R=3.3.1;CONDA_NPY=110;CONDA_GMP=5.1;CONDA_GSL=1.16;CONDA_BOOST=1.60;CONDA_NCURSES=5.9
Traceback (most recent call last):
  File "./simulate-travis.py", line 150, in <module>
    sp.run(['scripts/travis-run.sh'], env=env, universal_newlines=True, check=True)
  File "/sebastian/conda/conda3-build/bioconda/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['scripts/travis-run.sh']' returned non-zero exit status 1

Docker Containers for Reproducible Research Workshop (C4RR)

Hi @bioconda/core

There is this workshop coming:
https://www.software.ac.uk/c4rr

and I thought it could be interesting to present the bioconda project. What do you think?

Best regards,
Sebastian

export travis env vars to docker

xref: bioconda/bioconda-recipes#3271

tests

I think it's pretty clear that we need tests for the build system to avoid crippling the main bioconda-recipes repo. Quicker-running tests here will also allow faster development iterations than trying to test full bioconda-recipes PRs.

I'm thinking of setting up several small recipe dirs in this repo, a corresponding bioconda-utils channel on anaconda that can be pre-populated and used for testing for existing packages, and possibly a bioconda-utils docker hub account for at least tinkering with docker containers. The latter is so it will be isolated from the main bioconda-recipes repo while we work out how best to build.

@johanneskoester would you mind setting up a bioconda-utils channel and possibly a dockerhub account for this? For now we can copy the existing .travis.yaml structure (with subdags and encrypted env vars), so "setup" would mean committing a .travis.yaml with the new keys for this repo and conda channel, and pointing dockerhub to a Dockerfile in this repo.

package information not updated

I have pushed a new version of our tool intarna (v2.0.0) to bioconda two days ago bioconda-recipes pull 3720.

Unfortunately, the generated page is not (automatically) updating the version information at bioconda.github.io:

https://bioconda.github.io/recipes/intarna/README.html

(should be done every 24h right?)

Guess this is a bug and no feature.. ;)

Thanks for looking into this!

lint skip uses_setuptools

Hello,

I am trying to skip an unnecessary test here:
https://travis-ci.org/CGATOxford/bioconda-recipes/jobs/231235853#L602

by using [lint skip uses_setuptools for recipes/cgat-pipelines-nosetests] in the commit message but it does not seem to work.

Could you please help?

Best regards,
Sebastian

Simplify debugging recipe builds via `bioconda-utils build-debug`

Since bioconda-utils build invokes conda build in a subprocess, this is mainly feature tracking conda/conda-build#1751.

The change here would be simple when it is supported in conda-build.

ImportError: No module named 'conda_build'

Hi, so I'm trying to set up my environment, and I thought I did everything correctly.

I installed conda to ~/src/miniconda3, added ~/src/miniconda3/bin to the end of my PATH, and installed conda-build via
$ conda install conda-build

Then I installed bioconda-utils via
$ pip install --user git+https://github.com/bioconda/bioconda-utils.git

I checked that conda_build is installed via

$ source ~/src/miniconda3/bin/activate
(root) $ python3 -m conda_build
/home/me/src/miniconda3/bin/python3: No module named conda_build.__main__; 'conda_build' is a package and cannot be directly executed

But I still get this error when I try to run bioconda-utils:

(root) $ bioconda-utils -h
Traceback (most recent call last):
  File "/home/me/.local/bin/bioconda-utils", line 9, in <module>
    load_entry_point('bioconda-utils==0.9.0', 'console_scripts', 'bioconda-utils')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/me/.local/lib/python3.5/site-packages/bioconda_utils/cli.py", line 16, in <module>
    from . import utils
  File "/home/me/.local/lib/python3.5/site-packages/bioconda_utils/utils.py", line 22, in <module>
    from conda_build import api
ImportError: No module named 'conda_build'

FYI, here are details about my environment:

(root) $ conda list
# packages in environment at /home/me/src/miniconda3:
#
beautifulsoup4            4.5.3                    py36_0  
cffi                      1.9.1                    py36_0  
chardet                   2.3.0                    py36_0  
conda                     4.3.11                   py36_0  
conda-build               2.1.4                    py36_0  
conda-env                 2.6.0                         0  
conda-verify              2.0.0                    py36_0  
cryptography              1.7.1                    py36_0  
filelock                  2.0.7                    py36_0  
idna                      2.2                      py36_0  
jinja2                    2.9.4                    py36_0  
libffi                    3.2.1                         1  
markupsafe                0.23                     py36_2  
openssl                   1.0.2k                        0  
patchelf                  0.9                           0  
pip                       9.0.1                    py36_1  
pkginfo                   1.4.1                    py36_0  
pyasn1                    0.1.9                    py36_0  
pycosat                   0.6.1                    py36_1  
pycparser                 2.17                     py36_0  
pycrypto                  2.6.1                    py36_4  
pyopenssl                 16.2.0                   py36_0  
python                    3.6.0                         0  
pyyaml                    3.12                     py36_0  
readline                  6.2                           2  
requests                  2.12.4                   py36_0  
ruamel_yaml               0.11.14                  py36_1  
setuptools                27.2.0                   py36_0  
six                       1.10.0                   py36_0  
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
wheel                     0.29.0                   py36_0  
xz                        5.2.2                         1  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3

(root) $ conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.11
       conda is private : False
      conda-env version : 4.3.11
    conda-build version : 2.1.4
         python version : 3.6.0.final.0
       requests version : 2.12.4
       root environment : /home/me/src/miniconda3  (writable)
    default environment : /home/me/src/miniconda3
       envs directories : /home/me/src/miniconda3/envs
                          /home/me/.conda/envs
          package cache : /home/me/src/miniconda3/pkgs
                          /home/me/.conda/pkgs
           channel URLs : https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/r/linux-64
                          https://conda.anaconda.org/r/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
            config file : /home/me/.condarc
           offline mode : False
             user-agent : conda/4.3.11 requests/2.12.4 CPython/3.6.0 Linux/4.4.0-62-generic debian/stretch/sid glibc/2.23
                UID:GID : 1000:1000

(root) $ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
beautifulsoup4 (4.5.3)
cffi (1.9.1)
chardet (2.3.0)
conda (4.3.11)
conda-build (2.1.4)
conda-verify (2.0.0)
cryptography (1.7.1)
filelock (2.0.7)
idna (2.2)
Jinja2 (2.9.4)
MarkupSafe (0.23)
pip (9.0.1)
pkginfo (1.4.1)
pyasn1 (0.1.9)
pycosat (0.6.1)
pycparser (2.17)
pycrypto (2.6.1)
pyOpenSSL (16.2.0)
PyYAML (3.12)
requests (2.12.4)
setuptools (27.2.0)
six (1.10.0)
wheel (0.29.0)

conda-build 2.1.9 does not always render PKG_BUILDNUM

When obtaining the package path, conda-build 2.1.9 does not always render PKG_BUILDNUM.
This happens e.g. with pindel: conda-bld/linux-64/pindel-0.2.5b8-htslib1.4_.tar.bz2.

Since bioconda-utils relies heavily on this (e.g. it cannot properly filter such recipes and it cannot determine if the build succeeded), we have to wait with updating conda-build until this is fixed. For now, PR #85 reverts the updated pinning from #84.

TODO

create test case recipe
submit PR to conda-build with test case and fix

a working conda-build with docker

We were previously running into chown/permission issues when trying to run a docker container from inside a python script (see this issue in the docker repo for examples of others hitting the same problem).

The idea was to have a docker container do all the building, but the built packages would be placed in the host's conda-bld dir. That is, a conda build $recipe command that used a CentOS5/6 container to build, independent of the host OS.

I wanted to start simple, so I created a new repo at https://github.com/daler/build-test, which is now working on travis-ci and does exactly this. In fact, conda-build is not even installed on the travis-ci instance, and conda is only installed to test that the package is conda-installable. The slightly hacky part is building a new, local container each time based on the uid:gid of the user running the script. But it's fast, and it works.

So @johanneskoester, @bgruening if you guys think this is still a useful thing for bioconda-utils I can work on porting it over.

Missing README explaining how to update the website

This repository needs a minimal README saying where the content comes from, and how it is updated.

I was intending to submit a pull request to clarify the Java guidelines (say explicitly to add a note section to meta.yml giving the name of the wrapper script and details about the memory settings).

However, looking at the commit history it appears bioconda.github.io is only updated automatically - I presume from another repository?

use a patched condarc

As described in this conda issue, solver hangups can be avoided by modifying .condarc. This problem may be happening in bioconda/bioconda-recipes#5926, but even if not, we should have an easy and robust way of modifying a user's condarc to meet the channel order criteria.

Options for modifying:

curl $github_url > ~/.condarc is the easiest but also most disruptive to existing condarc
bioconda-utils setup-condarc can parse the existing yaml and update as needed (or warn about potential problems)
some sort of curl $url | bash thing that would, using bash, fix the condarc. Or maybe we can stick a minimal yaml parser into a single python module, and use that? Ideally it would be a no-dependency script.

@bioconda/core any preferences?

Update pinned conda-build version?

I was wondering if it would be possible to update the conda-build version to >=2.0.11? I'm working on a recipe and hitting a bug that was fixed in v2.0.11 which causes the build to fail.

Mullted-tests: fix quoting

in bioconda/bioconda-recipes#5515, mulled-build was failing because of a test in meta.yaml using single rather than double quotes.

conda-defaults and conda-forge binary incompatible

Following up on:
ContinuumIO/anaconda-issues#1403

Copy and paste:

The only possible resolution, while conda-defaults and coda-forge remain binary incompatible is not to mix them in the same environment

Should conda-forge still be encouraged as part of the setup for bioconda in the documentation?
http://bioconda.github.io/contribute-a-recipe.html#use-your-new-recipe

Best regards,
Sebastian

ModuleNotFoundError: No module named 'github'

Hello.

I have installed Miniconda3, and added it to my path. Installed conda-build, and followed the pip install from github command in the README.

When typing bioconda-utils.py -h, I get:

File "/home/agoncalves/miniconda3/lib/python3.6/site-packages/bioconda_utils/github_integration.py", line 1, in <module>
    import github
ModuleNotFoundError: No module named 'github'

Thank you.

Anders.

PS: Here is some environment information:

agoncalves@marvin:mlst $ conda info
    Current conda install:

                   platform : linux-64
              conda version : 4.3.17
           conda is private : False
          conda-env version : 4.3.17
        conda-build version : 2.1.12
             python version : 3.6.0.final.0
           requests version : 2.12.4
           root environment : /home/agoncalves/miniconda3  (writable)
        default environment : /home/agoncalves/miniconda3
           envs directories : /home/agoncalves/miniconda3/envs
                          /home/agoncalves/.conda/envs
              package cache : /home/agoncalves/miniconda3/pkgs
                          /home/agoncalves/.conda/pkgs
               channel URLs : https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
                config file : /home/agoncalves/.condarc
               offline mode : False
                 user-agent : conda/4.3.17 requests/2.12.4 CPython/3.6.0 Linux/3.10.0-514.16.1.el7.x86_64 Red Hat Enterprise Linux Server/7.3 glibc/2.17
                    UID:GID : 1424832727:1424491009

agoncalves@marvin:mlst $ conda list
# packages in environment at /home/agoncalves/miniconda3:
#
argh                      0.26.2                    <pip>
beautifulsoup4            4.5.3                    py36_0
bioconda-utils            0.10.0                    <pip>
blast                     2.6.0               boost1.61_0    bioconda
boost                     1.61.0                   py36_0
cffi                      1.9.1                    py36_0
chardet                   3.0.2                    py36_0
colorlog                  2.10.0                    <pip>
conda                     4.3.17                   py36_0
conda-build               2.1.12                   py36_0
conda-env                 2.6.0                         0
conda-verify              2.0.0                    py36_0
cryptography              1.7.1                    py36_0
decorator                 4.0.11                    <pip>
filelock                  2.0.7                    py36_0
icu                       54.1                          0
idna                      2.2                      py36_0
jinja2                    2.9.6                    py36_0
jsonschema                2.6.0                     <pip>
libffi                    3.2.1                         1
libgcc                    5.2.0                         0
markupsafe                0.23                     py36_2
networkx                  1.11                      <pip>
numpy                     1.12.1                    <pip>
openssl                   1.0.2k                        0
pandas                    0.20.1                    <pip>
patchelf                  0.9                           0
pip                       9.0.1                    py36_1
pkginfo                   1.4.1                    py36_0
pyasn1                    0.1.9                    py36_0
pycosat                   0.6.1                    py36_1
pycparser                 2.17                     py36_0
pycrypto                  2.6.1                    py36_4
pydotplus                 2.0.2                     <pip>
pyopenssl                 16.2.0                   py36_0
pyparsing                 2.2.0                     <pip>
python                    3.6.0                         0
python-dateutil           2.6.0                     <pip>
pytz                      2017.2                    <pip>
pyyaml                    3.12                     py36_0
readline                  6.2                           2
requests                  2.12.4                   py36_0
ruamel_yaml               0.11.14                  py36_1
setuptools                27.2.0                   py36_0
six                       1.10.0                   py36_0
sqlite                    3.13.0                        0
tk                        8.5.18                        0
wheel                     0.29.0                   py36_0
xz                        5.2.2                         1
yaml                      0.1.6                         0
zlib                      1.2.8                         3

Automatically add cargo-port URLs to recipes

Once a recipe gets archived, it should be updated to reflect the new URL (at least as a backup).

Maybe a separate archive module, which could also check for and handle bioconductor URL degradation.

xref bioconda/bioconda-recipes#1015 (comment)

CONDA_GSL issues

There seems to be an issue with GSL in bioconda at the moment. My package msprime depends on GSL and uses the CONDA_GSL variable. This worked well previously. Now though, if I delete the gsl package and run conda install msprime I get

$ conda install msprime
Fetching package metadata .............
Solving package specifications: .
Error: Dependency missing in current osx-64 channels: 
  - msprime -> gsl 1.16*

You can search for packages on anaconda.org with

    anaconda search -t conda gsl

You may need to install the anaconda-client command line client with

    conda install anaconda-client

This is on OSX.

The problem is perhaps to do with the fact the conda-forge is now using gsl version 2.1:

https://anaconda.org/conda-forge/gsl

I guess I have two questions:

Why do we use CONDA_GSL and pin the version rather than just having an ordinary dependency on gsl (I just followed the pattern I found in other recipes here, monkey-see-monkey-do)?
Should we update CONDA_GSL to 2.1 and trigger rebuilds of all the packages that use it?

Thanks to @molpopgen for pointing this problem out to me.

Re-enable subdags, but in a dynamic way

It would be nice to use multiple build jobs (remember our SUBDAG feature) in case of bigger pull requests. However, for the regular pull request, this is not necessary and creates a lot of overhead. We should try to do the following:

the regular, automatic travis job investigates the DAG of packages that need to be build.
if the DAG is small, go on with building it
if the DAG is big, and can be split up into several disconnected subdags, trigger a new travis build (see API) with an appropriate build matrix set (e.g. defining n SUBDAG jobs). Then stop the current job.

This way, we get parallelism for connected subdags in case of large builds, and don't have additional overhead in case of small builds. If this works, it is a much superior solution compared to the feedstock approach, because we are able to automatically deal with the dependencies between packages without the downside that all builds run in a single job.

@bioconda/core, what do you think?

colorlog install error !

Hi, I am struggled to build my package using the bioconda but encountered the following error:

Could not connect to https://conda.anaconda.org/conda-forge/linux-64/pygithub-1.29-py35_0.tar.bz2
An unexpected error has occurred.

Please consider posting the following information to the
conda GitHub issue tracker at:

https://github.com/conda/conda/issues

Current conda install:

           platform : linux-64
      conda version : 4.2.13
   conda is private : False
  conda-env version : 4.2.13
conda-build version : 2.1.10
     python version : 3.5.3.final.0
   requests version : 2.13.0
   root environment : /opt/conda  (writable)
default environment : /opt/conda
   envs directories : /opt/conda/envs
      package cache : /opt/conda/pkgs
       channel URLs : https://conda.anaconda.org/bioconda/linux-64
                      https://conda.anaconda.org/bioconda/noarch
                      https://conda.anaconda.org/r/linux-64
                      https://conda.anaconda.org/r/noarch
                      https://repo.continuum.io/pkgs/free/linux-64
                      https://repo.continuum.io/pkgs/free/noarch
                      https://repo.continuum.io/pkgs/pro/linux-64
                      https://repo.continuum.io/pkgs/pro/noarch
                      https://conda.anaconda.org/conda-forge/linux-64
                      https://conda.anaconda.org/conda-forge/noarch
        config file : /root/.condarc
       offline mode : False

$ /opt/conda/bin/conda install --file /tmp/requirements.txt

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 146, in _new_conn
    (self.host, self.timeout))
requests.packages.urllib3.exceptions.ConnectTimeoutError: (<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5792a29320>, 'Connection to binstar-cio-packages-prod.s3.amazonaws.com timed out. (connect timeout=6.1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 678, in urlopen
    **response_kw)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 678, in urlopen
    **response_kw)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 678, in urlopen
    **response_kw)
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/conda/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 376, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Max retries exceeded with url: /564af036eaad370b166e98f3/57fabe791cb1e3059c70b922?response-content-disposition=attachment%3B%20filename%3D%22pygithub-1.29-py35_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pygithub-1.29-py35_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20170525T085113Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FQoDYXdzEOf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDO%2Fed%2BGkVs7n5WW5OCK3A%2FYS%2F66OxdFjGkuqOteuunait2mEZoATYPLOyxL%2FkVKWTLhrUeOZkdmVoXUGxWW3%2FU8ITM7g9KPntPRjmcokcj1sPOnPAMetTYxsECLUsbYXRns9KmxGYMUYKc0ZUVfuJEJ5xQFMNn7b%2FnEZpxqbWiltnnIi5nZEjW8melnWy%2FSHP6eTk8dRt8KCrNQEdWlEbpIkdcTZTD00PvLnPj0ki7OlFqtRMQ1ZMJ%2BQZXgPMn8zl0meBVn4ndsrOKt%2BXQH6rI%2B5Ags2eegqEcDXdI6BENgvyv7ZBP%2BQ3VxObAYyKRlh8SN1jFc5z2vU%2BhCgzTRbthxnwZfg59DRu5%2F0KuPEdOuVISXjBv9yf2cEyajpJb0aJVDv0eFlADNmThUhV8DPD0yg7fSnNuScZQjKlWUB7iZ4b9le%2FaTwYLeaoWZrryYeNLicqWu%2FaIII6n4u0UqpMxMQyM3VlRq29wwAMctYt8eA7%2F2Nj3O2oqMs4OXCVks%2BneDg0etynSLFBepAfDEf16z%2Fd5HoPPnrVE%2FEa6amobp%2Ff6pMqMz46G6S7NSnp0Net1noe%2BbQeUAA3lNVCo3ktZ0f4DdkQA8oxeuZyQU%3D&X-Amz-Credential=ASIAJYEW2A5QVP3XICNA%2F20170525%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=2fd098bad4122a9be4adbf36e556966d5bbfa13719a7d69054cd07d7f8050476 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5792a29320>, 'Connection to binstar-cio-packages-prod.s3.amazonaws.com timed out. (connect timeout=6.1)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/conda/fetch.py", line 394, in download
    resp = session.get(url, stream=True, proxies=session.proxies, timeout=(6.1, 60))
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 630, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 630, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 190, in resolve_redirects
    **adapter_kwargs
  File "/opt/conda/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/requests/adapters.py", line 479, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Max retries exceeded with url: /564af036eaad370b166e98f3/57fabe791cb1e3059c70b922?response-content-disposition=attachment%3B%20filename%3D%22pygithub-1.29-py35_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pygithub-1.29-py35_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20170525T085113Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FQoDYXdzEOf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDO%2Fed%2BGkVs7n5WW5OCK3A%2FYS%2F66OxdFjGkuqOteuunait2mEZoATYPLOyxL%2FkVKWTLhrUeOZkdmVoXUGxWW3%2FU8ITM7g9KPntPRjmcokcj1sPOnPAMetTYxsECLUsbYXRns9KmxGYMUYKc0ZUVfuJEJ5xQFMNn7b%2FnEZpxqbWiltnnIi5nZEjW8melnWy%2FSHP6eTk8dRt8KCrNQEdWlEbpIkdcTZTD00PvLnPj0ki7OlFqtRMQ1ZMJ%2BQZXgPMn8zl0meBVn4ndsrOKt%2BXQH6rI%2B5Ags2eegqEcDXdI6BENgvyv7ZBP%2BQ3VxObAYyKRlh8SN1jFc5z2vU%2BhCgzTRbthxnwZfg59DRu5%2F0KuPEdOuVISXjBv9yf2cEyajpJb0aJVDv0eFlADNmThUhV8DPD0yg7fSnNuScZQjKlWUB7iZ4b9le%2FaTwYLeaoWZrryYeNLicqWu%2FaIII6n4u0UqpMxMQyM3VlRq29wwAMctYt8eA7%2F2Nj3O2oqMs4OXCVks%2BneDg0etynSLFBepAfDEf16z%2Fd5HoPPnrVE%2FEa6amobp%2Ff6pMqMz46G6S7NSnp0Net1noe%2BbQeUAA3lNVCo3ktZ0f4DdkQA8oxeuZyQU%3D&X-Amz-Credential=ASIAJYEW2A5QVP3XICNA%2F20170525%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=2fd098bad4122a9be4adbf36e556966d5bbfa13719a7d69054cd07d7f8050476 (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5792a29320>, 'Connection to binstar-cio-packages-prod.s3.amazonaws.com timed out. (connect timeout=6.1)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/conda/cli/install.py", line 405, in install
    execute_actions(actions, index, verbose=not context.quiet)
  File "/opt/conda/lib/python3.5/site-packages/conda/plan.py", line 643, in execute_actions
    inst.execute_instructions(plan, index, verbose)
  File "/opt/conda/lib/python3.5/site-packages/conda/instructions.py", line 135, in execute_instructions
    cmd(state, arg)
  File "/opt/conda/lib/python3.5/site-packages/conda/instructions.py", line 47, in FETCH_CMD
    fetch_pkg(state['index'][arg + '.tar.bz2'])
  File "/opt/conda/lib/python3.5/site-packages/conda/fetch.py", line 353, in fetch_pkg
    download(url, path, session=session, md5=info['md5'], urlstxt=True)
  File "/opt/conda/lib/python3.5/site-packages/conda/fetch.py", line 405, in download
    raise CondaRuntimeError(msg)
conda.exceptions.CondaRuntimeError: <unprintable CondaRuntimeError object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
    return_value = func(*args, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/conda/cli/main.py", line 145, in _main
    exit_code = args.func(args, p)
  File "/opt/conda/lib/python3.5/site-packages/conda/cli/main_install.py", line 80, in execute
    install(args, parser, 'install')
  File "/opt/conda/lib/python3.5/site-packages/conda/cli/install.py", line 420, in install
    raise CondaRuntimeError('RuntimeError: %s' % e)
  File "/opt/conda/lib/python3.5/site-packages/conda/__init__.py", line 45, in __str__
    return text_type(self.message % self._kwargs)
ValueError: unsupported format character 'B' (0x42) at index 238

It seems like the colorlog is not installed correctly, then I run the following command:

docker run condaforge/linux-anvil /opt/conda/bin/conda install colorlog=2.10.*

Then same error occurs. Then I run the following command:

docker run condaforge/linux-anvil /opt/conda/bin/conda install colorlog

It works. Can you check this error and modify your bioconda_utils-requirements.txt.
Because I can't modify the file on "travis-ci" website .
Hope you can understand what I said.

Provide documentation on how to build bioconda binaries for local / private use

I would like to be able to build a substantial portion of the bioconda binaries for use on a Centos5 system that cannot be upgraded to Centos6. Would it be possible to use bioconda-utils for this? And if so could such a use case be documented?

Lint: handle malformed yaml

xref: bioconda/bioconda-recipes#4723

Missing `should_be_noarch` documentation

Documentation for should_be_noarch is missing from https://bioconda.github.io/linting.html . The linting function was introduced in #134 .

Mulled-tests: honor run_test.sh & co

When preparing the shell script to test a built package, get_tests() in bioconda_utils/pkg_test.py only processes tests defined in the test: section of meta.yaml, but according to https://conda.io/docs/building/meta-yaml.html#run-test-script

The script run_test.sh (or .bat/.py/.pl) will be run automatically if it is part of the recipe.

conda build of a recipe containing a run_test.py confirms it.

recipes with no tests should automatically fail

xref: bioconda/bioconda-recipes#124 (comment)

Check pins in linter

There are a number of packages with versions known to break things. While they can't be removed, the linter can check that they are excluded via pinning by any updated recipe.

A good start would be the list hidden away in conda-forge's pin_the_slow_way.py. Adding to that, there are broken bioconda packages as well, such as sed 4.2.0 (pin to 4.4.*).

skeletons

bioconda-recipes/scripts/bioconductor/bioconductor_skeleton.py should be moved here, and incorporated into a larger skeleton module that includes perl, java, and c/c++ skeletons consistent with bioconda conventions.

xref bioconda/bioconda-recipes#162

linting issues

Tracking some TODOs as problems arise:

OS-specific version detection: If linux finishes and upload before OSX (which happens frequently), the OSX run complains that it already exists (like here). Solution is to temporarily disable the "already_in_bioconda" lint, which is done here

We need to simulate git commit history in the tests to better test utils.modified_recipes.

A few motivated edits to these docs

Coming from the perspective of a new (wannabe) contributor...

Install conda and Docker (one-time setup) should advise to conda install conda-build. There is NO MENTION of conda-build in this site. Failng to install it generates an conda: error: argument command: invalid choice, which is disheartening to the newb (me), despite the advice that appears at the end of the usage reminder, that other commands, such as "conda build", are available when additional conda packages (e.g. conda-build) are installed. It would be far better to advise the new contributor to install this package as part of contrib-setup.
Until this conda/conda-build#316 fixed, newbs to this project such as myself would really appreciate a warning to expect this in Test locally
One-time setup should advise to cd bioconda-recipes prior to git remote add upstream https://github.com/bioconda/bioconda-recipes.git

All in favor? Should I take the plunge?

bioconda package for bioconda-utils?

Would it make sense to have a conda package for bioconda-utils, so it can be installed as easily as conda-build? If we make one, the recipe could include anaconda-verify as a dependency to support #19.

Skip lints in extra section of recipe

It would be nice if we could persist lint skipping in the recipe, via the extra section. E.g.:

extra:
  skip-lints:
      - uses_git_url
      - missing_hash

@daler, what do you think?

'locale' missing in Docker image?

Using bioconda-utils as a site specific package builder

pinging @bioconda/core

This is mostly just a shout out for the great work you've done on infrastructure. ;-)

We are going to be using the bioconda-utils package and build system (including mulled builds and whatnot), as a way of building our analysis/lab/project packages, which are just meta packages of bioconda and sometimes R packages. Initially we were just using conda-envs, but that has proven to be problematic if things change upstream, so instead, we will just build packages and leave them there.

I've been taking a look at the code, and it looks likes its ready for being site specific. I think I should only have to change some of the variables in the scripts/travis-run.sh script, and of course the .travis needs to have our own github token and things, but I will let you know if you run into any snags.

Travis builds are skipped based on host os, not TRAVIS_OS_NAME

If a recipe specifies it should be skipped on OSX, and I run TRAVIS_OS_NAME=linux ./simulate-travis.py --docker --packages quast I expect that the test will be run on Linux. However, instead, no tests are run:

DEBUG bioconda_utils.utils:tobuild(486): BIOCONDA FILTER: not building quast-4.3-boost1.60_0.tar.bz2 because it defines skip for this env
DEBUG bioconda_utils.utils:tobuild(486): BIOCONDA FILTER: not building quast-4.3-boost1.60_0.tar.bz2 because it defines skip for this env
DEBUG bioconda_utils.utils:tobuild(486): BIOCONDA FILTER: not building quast-4.3-boost1.60_0.tar.bz2 because it defines skip for this env

If I remove the # [not osx] from meta.yaml and run the same command, the linux build is attempted.

Add `conda update --all` to installation instructions

May I suggest that conda update --all is added to the channel setup instructions on the Bioconda homepage. This would minimize the risk of users running into binary incompatibility. This just happened to me with the default readline package and the r package after a fresh Miniconda install.

You can close this issue if you don't like the idea. I just wanted to share some feedback to help future Bioconda users. Cheers!

Making mulled-build optional

I was trying to build an OS X package and running simulate-travis.py ended up failing due to trying to perform a mulled-build. Since I don't need mulled build functionality, is it possible to make mulled-build optional per package? We can still have it be true by default but have an optional to turn it off.

Add `conda verify` to test pipeline

https://www.continuum.io/blog/developer-blog/verifying-conda-recipes-and-packages-anaconda-verify

disable uploading if ANACONDA_TOKEN does not exist

To avoid issues when testing a fork on the fork's master branch, xref bioconda/bioconda-recipes#3724

Extra channel list not passed correctly to mulled-build

I noticed the mulled-build test was failing when testing a package that relied on dependencies from conda-forge. Looking closer, the extra channel arguments in the test_package function of pkg_test.py are not formatted correctly. The relevant snippet is:

for channel in channels:
    channel_args.extend(['--extra-channel', channel])

First, there's a typo there. The argument name mulled-build accepts is --extra-channels; the trailing s is missing. More important though, is that mulled-build expects a comma-separated list of channels and not multiple instances of the argument. If I change the above to

channel_args.extend(['--extra-channels', ','.join(channels)])

then the channel list gets passed correctly and the test succeeds.

Document some tricky details

I discovered a couple of tricky bits:

Need python 3 presumably (see stacktrace below)
You have to pre-install conda-build, because otherwise pip will try to do it automatically and choke. (see logs below)
The root python environment needs to be python3, and you need to be installing into the root env.

yield from map(os.path.dirname,
             ^
SyntaxError: invalid syntax

LOCKERROR: It looks like conda is already doing something.

Having this problem here:
https://travis-ci.org/bioconda/bioconda-recipes/jobs/206522192#L576
https://travis-ci.org/bioconda/bioconda-recipes/jobs/206536065#L480

I have reported to conda:
conda/conda#3981

Conda environment variables not exported to container

Hello,

I have initially reported the issue at bioconda-recipes:
bioconda/bioconda-recipes#3780

but I think this is the correct forum.

I have a new recipe which uses SRC_DIR to run tests. SRC_DIR is a conda environment variable:
https://conda.io/docs/building/environment-vars.html

but it is not available when running simulate-travis.py.

Is this the intented behaviour?

Many thanks,
Sebastian

linters

Need linters for existing recipes to flag easily-identified issues.

For example, if gcc is a dependency, then llvm should be a dependency for osx and libgcc should be a run dependency.

I'm picturing this as a module of functions that each take a recipe and check for a discrete issue, returning a list of issues identified. Adding new functionality to the linter would be a matter of just adding a new function.

Mulled-Tests: permission denied during tmpdir remove after mulled-build

TLDR: with tempfile.TemporaryDirectory() does not force delete but fails on permission denied.

I've got a recipe (megahit bioconda/bioconda-recipes#5877) failing the mulled-test because of a test result file (final.contigs.fa) that can't be removed when leaving the with tempfile.TemporaryDirectory() as d at the bottom of pkg_test.py (log, traceback at bottom).

The exception occurs inside of TemporaryDirectory.cleanup() in shutil.rmtree() due to permission denied while trying to unlink a file inside of a write protected directory. (Or perhaps wrong user, as a result of the docker execution).

shutil.rmtree, and by extension the with TemporaryDirectory(), only do the equivalent of rm -r and provide no easy option to do a rm -rf. Here's some sample code to reproduce:

import tempfile, os, stat
for perm in 0, stat.S_IREAD|stat.S_IEXEC:
 with tempfile.TemporaryDirectory() as d:
  os.mkdir(os.path.join(d, "tmp"))
  with open(os.path.join(d, "tmp", "file"), "w") as f:
    f.write("test")
    os.chmod(os.path.join(d, "tmp"), perm)

Fixing this in a pythonic way is tedious. Bare code:

@contextlib.contextmanager
def TempDir():
  tmpdir = tempfile.TemporaryDirectory()
  yield tmpdir.name

  def onerr(func, path, _):
      if func in (os.listdir, os.open):
          os.chmod(path, stat.S_IWRITE|stat.S_IREAD|stat.S_IEXEC)
          # there is no way to return the listing or fp to shutil.rmtree, so we have to
          # delete the contents of the failed directory
          shutil.rmtree(path, onerror=onerr)
          if func == os.listdir:
              # os.listdir is used in the unsafe (not using fd) path and unlike the safe
              # fd (version) will try to unlink the directory itself after failure. so we
              # give it one:
              os.mkdir(path)
      else: # func in (os.unlink, os.islink, os.open(dir), os.rmdir
          os.chmod(os.path.dirname(path), stat.S_IWRITE|stat.S_IREAD|stat.S_IEXEC)
          func(path)

   shutil.rmtree(tmpdir.name, onerror=onerr)

Considering how pretty the above is, I'd go for a KISS solution and use the tried and tested rm -rf (which will try it's best to get rid of everything, but never fail).

@contextlib.contextmanager
def TempDir():
  tmpdir = tempfile.TemporaryDirectory()
  yield tmpdir.name
  sp.run(['rm', '-rf', tmpdir.name])

Traceback:

Traceback (most recent call last):
  File "/anaconda/bin/bioconda-utils", line 11, in <module>
    load_entry_point('bioconda-utils==0.10.0', 'console_scripts', 'bioconda-utils')()
  File "/anaconda/lib/python3.5/site-packages/bioconda_utils/cli.py", line 537, in main
    argh.dispatch_commands([build, dag, dependent, lint, duplicates, bioconductor_skeleton])
  File "/anaconda/lib/python3.5/site-packages/argh/dispatching.py", line 328, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/anaconda/lib/python3.5/site-packages/argh/dispatching.py", line 174, in dispatch
    for line in lines:
  File "/anaconda/lib/python3.5/site-packages/argh/dispatching.py", line 277, in _execute_command
    for line in result:
  File "/anaconda/lib/python3.5/site-packages/argh/dispatching.py", line 260, in _call
    result = function(*positional, **keywords)
  File "/anaconda/lib/python3.5/site-packages/bioconda_utils/cli.py", line 410, in build
    mulled_upload_target=mulled_upload_target,
  File "/anaconda/lib/python3.5/site-packages/bioconda_utils/build.py", line 352, in build_recipes
    docker_builder=docker_builder,
  File "/anaconda/lib/python3.5/site-packages/bioconda_utils/build.py", line 164, in build
    res = pkg_test.test_package(pkg_path, base_image=base_image)
  File "/anaconda/lib/python3.5/site-packages/bioconda_utils/pkg_test.py", line 147, in test_package
    p = utils.run(cmd, env=env, cwd=d)
  File "/anaconda/lib/python3.5/tempfile.py", line 808, in __exit__
    self.cleanup()
  File "/anaconda/lib/python3.5/tempfile.py", line 812, in cleanup
    _shutil.rmtree(self.name)
  File "/anaconda/lib/python3.5/shutil.py", line 474, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/anaconda/lib/python3.5/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/anaconda/lib/python3.5/shutil.py", line 432, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/anaconda/lib/python3.5/shutil.py", line 430, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'final.contigs.fa'

Cannot handle a list for source field

I am building a recipe where I have multiple sources. After following the instructions below.
https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#source-from-multiple-sources

Getting the following error from lint.

Traceback (most recent call last):
File "/tmp/miniconda/bin/bioconda-utils", line 11, in
load_entry_point('bioconda-utils==0.10.0', 'console_scripts', 'bioconda-utils')()
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/cli.py", line 503, in main
argh.dispatch_commands([build, dag, dependent, lint, duplicates])
File "/tmp/miniconda/lib/python3.5/site-packages/argh/dispatching.py", line 328, in dispatch_commands
dispatch(parser, *args, **kwargs)
File "/tmp/miniconda/lib/python3.5/site-packages/argh/dispatching.py", line 174, in dispatch
for line in lines:
File "/tmp/miniconda/lib/python3.5/site-packages/argh/dispatching.py", line 277, in _execute_command
for line in result:
File "/tmp/miniconda/lib/python3.5/site-packages/argh/dispatching.py", line 260, in _call
result = function(*positional, **keywords)
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/cli.py", line 409, in build
mulled_upload_target=mulled_upload_target,
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/build.py", line 267, in build_recipes
recipes, env_matrix, check_channels, force=force)
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/utils.py", line 743, in filter_recipes
pkg = built_package_path(recipe, env)
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/utils.py", line 519, in built_package_path
path = api.get_output_file_path(meta, config=config)
File "/tmp/miniconda/lib/python3.5/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/utils.py", line 111, in temp_env
yield
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/utils.py", line 517, in built_package_path
meta = MetaData(recipe, config=config)
File "/tmp/miniconda/lib/python3.5/site-packages/conda_build/metadata.py", line 437, in init
self.parse_again(config=config, permit_undefined_jinja=True)
File "/tmp/miniconda/lib/python3.5/site-packages/conda_build/metadata.py", line 470, in parse_again
config=config, path=self.meta_path)
File "/tmp/miniconda/lib/python3.5/site-packages/conda_build/metadata.py", line 196, in parse
(field, res[field].class.name, path))
RuntimeError: The source field should be a dict, not list in file recipes/bcftools-snvphyl-plugin/meta.yaml.
Exception ignored in: <bound method RecipeBuilder.del of <bioconda_utils.docker_utils.RecipeBuilder object at 0x7fc02cba2ef0>>
Traceback (most recent call last):
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/docker_utils.py", line 367, in del
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/docker_utils.py", line 507, in cleanup
File "/tmp/miniconda/lib/python3.5/site-packages/bioconda_utils/utils.py", line 194, in run
File "/tmp/miniconda/lib/python3.5/subprocess.py", line 693, in run
File "/tmp/miniconda/lib/python3.5/subprocess.py", line 947, in init
File "/tmp/miniconda/lib/python3.5/subprocess.py", line 1481, in _execute_child
File "/tmp/miniconda/lib/python3.5/os.py", line 673, in get_exec_path
File "", line 969, in _find_and_load
File "", line 954, in _find_and_load_unlocked
File "", line 887, in _find_spec
TypeError: 'NoneType' object is not iterable

Looks like it does not like getting a list instead of a dictionary. Any ideas how to skip this? Is there a flag I can pass to ./simulate-travis.py or travis itself to ignore linting?

bioconda / bioconda-utils Goto Github PK

bioconda-utils's People

Contributors

Stargazers

Watchers

Forkers

bioconda-utils's Issues

TODO

Recommend Projects

Recommend Topics

Recommend Org