Why does conda-build set the LANG environment variable to en_US.UTF-8? Shouldn't it set it to os.getenv('LANG')?

We can look at using https://github.com/IQAndreas/github-issues-import to migrate the conda build issues over, if it isn't too much work.

If you have a package that modifies some file that is part of some other package, this modification is not bundled along with the conda package that gets built. This is because build.py only includes the difference in the set of files in the prefix before building a package and after building a package.

This may be intentional behavior, but it can be frustrating for things like Perl, where newer versions of core modules sometimes overwrite the included modules.

From the support list:

Hi,

I've added a few test cases to my build recipe. The test fail if the module is not installed in the current conda environment on the import of mymodule in test1. i.e. If "conda-build --no-tests" is ran and the generated module is installed in current environment, a following conda-build has successful tests.

in meta.yaml:

test:

Python imports

imports:
- mymodule

files:
- test/test1.py
- test/test2.py

commands:
# You can put test commands to be run here. Use this to test that the
# entry points work.
- python test1.py
- python test2.py

doing a conda remove mymodule in current environment makes the build tests fail again.

Should the module being built be available to the tests, and how then? now it seems like it imports it from the current environment.

See #13.

There needs to be a way for conda-build to automatically try and build any missing dependencies if it can match the dependency to a recipe directory. This functionality is essential for building the R cran-conda-recipes as a given recipe may have anywhere from 5-15 different dependencies that won't have been built yet.

I put in very hacky support in the original r branch via this commit:
conda/conda@efba6b5

Which I'd then make use of with github.com/tpn/cran-conda-recipes/rbuild:
https://github.com/tpn/cran-conda-recipes/blob/master/rbuild

Basically, I made conda write the name of the missing package to conda-missing-package.txt, then I'd read that after a conda build failure, and recursively keep building failed deps until it worked, then re-try failed deps again whilst unwinding the stack.

It was super hacky but it worked. The elegant solution would be to make the solver cognizant of build requirements, not just install-time requirements -- but I'd be happy with whatever gets me recursive build functionality again, ideally in the master/mainline.

When I use

{% extends "setuptools.yaml" %

In my meta.yaml

I get the following error:

  File "/home/robertl/anaconda/lib/python2.7/site-packages/conda_build/jinja_context.py", line 31, in load_setuptools
    exec(open('setup.py', encoding='utf-8').read())
  File "<string>", line 382, in <module>
  File "<string>", line 238, in build_description
NameError: global name 'log' is not defined

If I remove the log, which is imported as from distutils import log

Then I get

  File "/home/robertl/anaconda/lib/python2.7/site-packages/conda_build/templates/setuptools.yaml", line 2, in top-level template code
    {% block body -%}
  File "/home/robertl/anaconda/lib/python2.7/site-packages/conda_build/templates/setuptools.yaml", line 3, in block "body"
    {% block package -%}
  File "/home/robertl/anaconda/lib/python2.7/site-packages/conda_build/templates/setuptools.yaml", line 5, in block "package"
    name: {{data.get('name').lower().replace(' ', '_')}}

I Tried to understand the problem by writing a small test script the repeats what cond-build is doing:

import setuptools
from io import open

data={}
def setup(**kw):
        data.update(kw)
setuptools.setup=setup
exec(open('../setup.py', encoding='utf-8').read())
print data['name']

I called the script test.py

and used the Anaconda python to run it:

$ ./anaconda/bin/python test.py

However, it gives the correct output (on all versions of my setup.py)

Found executable /usr/bin/gfortran
Attempting to autodetect OpenMP support...
Compiler supports OpenMP
Attempting to autodetect OpenMP support...
Compiler supports OpenMP
blas_opt not found
ATLAS version 3.8.4 built by ilan on Mon Jul  9 23:32:04 CDT 2012:
   UNAME    : Linux centos5x86 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei1 -DATL_CPUMHZ=3296 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 524288
   F77      : gfortran, version GNU Fortran (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52)
   F77FLAGS : -O -fPIC -m64
   SMC      : gcc, version gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52)
   SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -fPIC -m64
   SKC      : gcc, version gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52)
   SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -fPIC -m64

Token {'libraries': ['f77blas', 'cblas', 'atlas'], 'library_dirs': ['/home/robertl/anaconda/lib'], 'define_macros': [('ATLAS_INFO', '"\\"3.8.4\\""')], 'language': 'c'}

arachnid

The info/recipe/meta.yaml file in a conda package may be invalid. Conda should check this before returning a successful build result.

From conda-archive/conda-recipes#98

For conda build to be useful in a CI setting, where builds are regularly created from a version control branch, the version number needs to be set after the build is finished. Maybe the version tag would be excluded from the build yaml, and a post_build_version tag or similar would be created, which is a script to execute on the built package to request its version number.

post_build_version=python -c 'import dynd;print(dynd.__version__)'

if a recipe has a build/run requirement of conda itself I get this Error: 'conda' can only be installed into root environment

e.g. conda-build requires 'conda'

if I remove 'conda' from the requirement I get:

Searching for conda
Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    install_requires = ['conda'],
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/distutils/dist.py", line 929, in run_commands
    self.run_command(cmd)
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/distutils/dist.py", line 948, in run_command
    cmd_obj.run()
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/install.py", line 74, in run
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/install.py", line 97, in do_egg_install
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/easy_install.py", line 358, in run
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/easy_install.py", line 574, in easy_install
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/easy_install.py", line 625, in install_item
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/easy_install.py", line 671, in process_distribution
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/pkg_resources.py", line 564, in resolve
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/pkg_resources.py", line 802, in best_match
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/pkg_resources.py", line 814, in obtain
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/command/easy_install.py", line 581, in easy_install
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/package_index.py", line 593, in fetch_distribution
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/package_index.py", line 431, in find_packages
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/package_index.py", line 771, in scan_url
  File "/home/workerm/miniconda3/envs/_build/lib/python3.3/site-packages/setuptools-2.1-py3.3.egg/setuptools/package_index.py", line 283, in process_url
RuntimeError: Setuptools downloading is disabled in conda build  url=https://pypi.python.org/simple/conda/r
+ exit
Command failed: /bin/bash -x -e /home/workerm/miniconda3/conda-recipes-own/conda_build/build.sh

May I ask how one does that in the correct manner?

It should match the Anaconda build system.

Since I upgraded to the latest version of conda (3.2.1) and conda-build (1.3.0), the following commands fail:

C:\> conda build coverage 
C:\> conda build coverage --test 
[...]
TypeError: environment can only contain strings

The problem is that conda_build/conda.py passes an env argument to subprocess.Popen, containing entries of invalid data types (e.g. unicode strings). Full stack trace:

C:> conda build coverage --test
TEST START: coverage-3.7-py33_1
Fetching package metadata: ....
Solving package specifications: ...
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    coverage-3.7               |           py33_1         186 KB

The following packages will be linked:

    package                    |            build
    ---------------------------|-----------------
    coverage-3.7               |           py33_1   hard-link
    python-3.3.5               |                0   hard-link

Fetching packages ...
coverage-3.7-py33_1.tar.bz2 100% |##################| Time: 0:00:00  27.18 MB/s
Extracting packages ...
[      COMPLETE      ] |#################################################| 100%
Linking packages ...
[      COMPLETE      ] |#################################################| 100%
Traceback (most recent call last):
  File "C:\Anaconda\Scripts\conda-build-script.py", line 4, in <module>
    sys.exit(main())
  File "C:\Anaconda\lib\site-packages\conda_build\main_build.py", line 76, in main
    args.func(args, p)
  File "C:\Anaconda\lib\site-packages\conda_build\main_build.py", line 189, in execut
    build.test(m)
  File "C:\Anaconda\lib\site-packages\conda_build\build.py", line 347, in test
    env=env, cwd=tmp_dir)
  File "C:\Anaconda\lib\subprocess.py", line 537, in check_call
    retcode = call(*popenargs, **kwargs)
  File "C:\Anaconda\lib\subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "C:\Anaconda\lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "C:\Anaconda\lib\subprocess.py", line 948, in _execute_child
    startupinfo)
TypeError: environment can only contain strings

I have binstar installed, but it can't find it. This is in a newly setup Ubuntu VM:

Error: cannot locate binstar (required for upload)
# Try:
# $ conda install binstar

aaronmeurer@ubuntu:~/Documents/conda-recipes(pyqt=)$binstar
usage: binstar [-h] [--show-traceback] [-t TOKEN] [-v] [-V]

               {auth,collections,config,groups,login,logout,package,publish,unpublish,remove,search,show,upload,whoami}
               ...
binstar: error: too few arguments
aaronmeurer@ubuntu:~/Documents/conda-recipes(pyqt=)$type binstar
binstar is hashed (/home/aaronmeurer/anaconda/envs/binstar/bin/binstar)

Here is the setup.py https://bitbucket.org/hpk42/pytest-xdist/src/b47ddaa7cc6ceb0345895db32f65861d450982f9/setup.py?at=default.

The entry points are entry_points = {'pytest11': ['xdist = xdist.plugin'],},. There is no pytest11 installed. It installs into py.test, so the preserve egg dir option has to be set.

when a source distribution is unpacked, its file path suffix is compared against a set of supported filename extensions. if the filename extension of the source distribution file name contains any uppercase letters the comparison fails and unpacking is delegated to the build script.

If we can pass an argument to conda build allowing an override to conda/conda-bld, then we should be able to have concurrent builds happen without the lock files getting in the way, and without needing a bunch of miniconda installs.

conda skeleton pypi oct2py creates import tests like oct2py/ipython (instead of oct2py.ipython).

A signature looks e.g. like:

http://cvc4.cs.nyu.edu/builds/src/cvc4-1.3.tar.gz.asc

and can be verified like this:

$ gpg --verify cvc4-1.3.tar.gz.asc  ~/conda-bld/src_cache/cvc4-1.3.tar.gz

I'm unclear on how to best implement this. Checking against a key requires e.g. importing said key via gpg --recv-keys. I don't know where in the process this should happen.

$conda --debug build xattr
Error: no such directory: /Users/aaronmeurer/Documents/Continuum/conda-recipes/build

I still hope you will add the ability to specify multi-processor-build option for increased compile speed without the need to rewrite the recipes

P

conda/install.py", line 398:
log.error('failed to link (src=%r, dst=%r, type=%r, error=%4)' %

There is no way to access $PREFIX in a generic way in the test commands, like there is in build.sh (or any of the other environment variables). Can we copy most of environ.py into the test header, replacing build_prefix with test_prefix, and include that in the env of check_call? We will also need to inline the actual value of test_prefix when in create_test.py.

Assumedly a common use-case of conda build --test is to test packages whose tests failed without rebuilding them. But if a recipe's tests fail, the package is placed in conda-bld/broken, where it won't be found by conda build --test.

I think it has something to do with the way conda build calls conda-build, but keyboard interrupting the build no longer deletes the lock file.

The recipe should be copied during the build. If something modifies the recipe during the build, it modifies the original recipe. This happens even for things like the new script parameter, which writes a build script to the recipe itself.

If the source download is incorrect, it will raise an exception that the checksums mismatch, but there doesn't seem to be a way to force it to redownload, other than manually deleting the source cache.

auto-remove linux .la files

if one builds more apps within the conda framework like gmp, mpfr, gcc, glibc and the whole essential development apps one might run into a number of issues
one of them might be: .la

example of conda build gmp .la

libgmp.la

...
...

# Directory that this library needs to be installed in:
libdir='/opt/anaconda1anaconda2anaconda3/lib'

one can read more for instance here: http://www.metastatic.org/text/libtool.html

to avoid removing them individually I did a small function in conda-build

$conda skeleton pypi cobra
Using url https://pypi.python.org/packages/source/c/cobra/cobra-0.2.1.tar.gz (1719270) for cobra.
Downloading cobra (use --no-download to skip this step)
Using cached download
Unpacking cobra...
done
working in /var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmpehwr8econda_skeleton
Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda/bin/conda-skeleton", line 12, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/bin/conda-skeleton", line 5, in <module>
    sys.exit(main())
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/main_skeleton.py", line 78, in main
    args.func(args, p)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/main_skeleton.py", line 85, in execute
    pypi.main(args, parser)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/pypi.py", line 246, in main
    run_setuppy(src_dir)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/pypi.py", line 381, in run_setuppy
    exec(compile(f.read(), 'setup.py', 'exec'), d)
  File "setup.py", line 1, in <module>
    #!/usr/bin/env python
ImportError: No module named 'ez_setup'

In #59 I get

$conda skeleton pypi cobra
Using url https://pypi.python.org/packages/source/c/cobra/cobra-0.2.1.tar.gz (1719270) for cobra.
Downloading cobra (use --no-download to skip this step)
Using cached download
Unpacking cobra...
done
working in /var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmpoxgw5jconda_skeleton
Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda/bin/conda-skeleton", line 12, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/bin/conda-skeleton", line 5, in <module>
    sys.exit(main())
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/main_skeleton.py", line 79, in main
    args.func(args, p)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/main_skeleton.py", line 86, in execute
    pypi.main(args, parser)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/pypi.py", line 273, in main
    run_setuppy(tempdir, src_dir)
  File "/Users/aaronmeurer/Documents/Continuum/conda-build/conda_build/pypi.py", line 406, in run_setuppy
    with open(join(src_dir, 'setup.py'), encoding='utf-8') as setuppy:
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmpoxgw5jconda_skeleton/setup.py'

This is primarily required for R, although it could very well be applicable to other projects or libraries that have a similar situation.

In order for R to be relocatable and to also work with our tcl/tk libs, TCL_CONFIG and TK_CONFIG environment variables need to be exported at runtime to point to the correct locations (which will be environment-dependent).

The only option now is to remember to alter your shell's startup scripts and manually export the necessary envvars. That's not a good long-term solution. (It took me forever to figure out that R needed those envvars set explicitly; we can't expect other users to know they need to be set.)

So, I think there's a need for a pre-activate.sh-type script that allows a package to customize an environment as part of 'activation'. This would also entail having the root environment automatically run any pre-activate.sh scripts, which isn't possible with our current (simple) .bashrc alteration.

When, a file post-link.sh (and pre-unlink.sh, and .bat versions) is present in the recipe directory, it should be automatically be included in the conda package at the right place.

I don't know if this is a conda-build issue or not.

It is a known conda issue that if there is a newer version of a package for a different Python, instead of choosing the older version for the Python you want, it will just die with "Unsatisfiable package specifications". The workaround when using conda is to set python=2.6 or whatever explicitly.

It seems the same issue shows up when using conda build with CONDA_PY. CONDA_PY should set the Python spec the same way, so that it picks the correct version of the package.

trying to build a pkg I get: Warning: 2 possible package resolutions:

but I do not find anyway to select one of them.

the one conda-build chooses is not the one I wanted.

This is a copy over from an issue I submitted against conda.

This has come up here, here, and here. Specifically, GDAL and pyOpenSSL at least require some things that should be passed in via --include-dirs and --library-dirs.

GDAL is now supported by conda, but pyopenssl is also / still a problem, and yeilds a confusing long stack trace when you try to install scrapy (which is used by "serious scientific computation folks").

I don't fully understand osx_is_app (and btw it's not documented anywhere), but if means what I think it does, the test imports should be run with pythonw, not python on OS X.

Now that conda-build is a separate project, we can make chrpath a dependency.

Suggestion from conda/conda#430:

1. Build: install option
   When one builds a package one gets the question:
   Do you want to upload this package to binstar ([y]/n)?

In the same manner I think it would be convenient to get before that an option like:
Do you want to install this package ([y]/n)?

because usually if one builds a new package one also wants to install it: at least an end user.

sometimes a package builder might want to set no rpath or a own one for internal usage.

it would be nice if one could have an option to disable the mk_relative rpath part.

build example something like

#CHECK: BUILD_NO_RPATH
if [ "${BUILD_NO_RPATH}" != "y" ]; then
    echo 'this recipes needs conda config: build_no_rpath: True'
    exit 1
fi

unset LD_RUN_PATH
./bootstrap.sh
LDFLAGS="-Wl,-rpath,${PREFIX}/usr" \
./configure --prefix=${PREFIX}
make
make install

Perhaps the os.chmod command should be wrapped in a try except statement

Traceback (most recent call last):
  File "/opt/anaconda/bin/conda", line 5, in <module>
    sys.exit(main())
  File "/opt/anaconda/lib/python2.7/site-packages/conda-2.2.3_24_g9caa576-py2.7.egg/conda/cli/main.py", line 186, in main
    args.func(args, p)
  File "/opt/anaconda/lib/python2.7/site-packages/conda-2.2.3_24_g9caa576-py2.7.egg/conda/cli/main_build.py", line 169, in execute
    build.build(m, pypi=args.pypi)
  File "/opt/anaconda/lib/python2.7/site-packages/conda-2.2.3_24_g9caa576-py2.7.egg/conda/builder/build.py", line 181, in build
    post_build(sorted(files2 - files1))
  File "/opt/anaconda/lib/python2.7/site-packages/conda-2.2.3_24_g9caa576-py2.7.egg/conda/builder/post.py", line 192, in post_build
    fix_permissions(files)
  File "/opt/anaconda/lib/python2.7/site-packages/conda-2.2.3_24_g9caa576-py2.7.egg/conda/builder/post.py", line 187, in fix_permissions
    os.chmod(path, stat.S_IMODE(st.st_mode) | stat.S_IWUSR) # chmod u+w
OSError: [Errno 1] Operation not permitted: '/var/lib/jenkins/envs/_build/lib/node_modules/cloud9/node_modules/jsDAV/node_modules/gnu-tools/bin/find'

hi @ilanschnell ,

just saw the separate repo: conda-build and I would be interested to know what are the plans for it.

are the plans to finally remove the build option from the conda package manager or just some additional tools.

I think to separate it would be a nice one: most people will only want to use conda as an installation tool for binary packages.

and if one wants to build a separate more specified package might be quite nice.

Anyway, looking forward to here from you what are the plans for it.

P

If the runtime dependencies of a package are unsatisfiable, at test time it will just say "unsatisfiable dependencies, hint: package has a conflict with the remaining packages", where package is the package you built. This isn't very helpful. It should test installing the dependencies independently, which will give a better hint.

Alternately (and perhaps this would be better), we could modify the hint code in conda to detect this case and give more useful information. If a package by itself is unsatisfiable we can give information about which of its dependencies conflict.

I'm not sure whether this is something to be solved within conda or the upstream binary. I tried building hyperdex which then doesn't start at a later point because it execs programs in /libexec/... :

failed to exec coordinator: No such file or directory
PATH=/home/teh/envs/_build/libexec/hyperdex-1.0.5:.:/usr/bin

Note the _build part when the full path should really be /home/teh/envs/testenv/ for a hypothetical testenv environment.

I guess programs like git would have the same problem though in git at least the libexec path is controllable via "GIT_EXEC_PATH" (which then needs to be changed in each conda environment).

Any ideas on how to solve this?

If you name a conda recipe that is a jinja template meta.yaml instead conda.yaml then binstar cannot upload the distribution.

It complains about the meta data.

Tracking this error back with --traceback shows that the meta.yaml jinja2 template is copied to the info directory, whereas if the file is named conda.yaml then a proper meta.yaml is built.

In short, a good workaround is to use conda.yaml filename with jinja2 build templates.

Hi,

While trying to troubleshoot some testing issues, I found that the path environment variable during test (printed from run_test.py) contains an unexpanded %PATH%

PATH = C:\Anaconda32\envs_test\Scripts;C:\Anaconda32\envs_test\Library\bin;C:\Anaconda32\envs_test;C:\Anaconda32\envs_test\Scripts;%PATH%

My PATH when executing the conda-build looks "normal".