conda-forge / conda-smithy Goto Github PK
View Code? Open in Web Editor NEWThe tool for managing conda-forge feedstocks.
Home Page: https://conda-forge.org/
License: BSD 3-Clause "New" or "Revised" License
The tool for managing conda-forge feedstocks.
Home Page: https://conda-forge.org/
License: BSD 3-Clause "New" or "Revised" License
The message about the spacing for selectors isn't clear to see the problem in the number of spaces:
Selectors are suggested to take a " # []" form.
It would be clearer to have a message like: <two spaces> # <one space> [selector]
In most cases: conda package name = pypi name, so it should be possible to check if there is a new release on pypi.
To help make it stand out, maybe we should colorize the linter text using something like termcolors
. Bare minimum would be to make fail red and pass green. Otherwise, we might want to colorize warnings or other relevant errors. Shouldn't be too bad, but will already be a huge asset for the review process.
We currently build against numpy:
We should drop 1.9 from the template.
More linter ideas for after the package is build:
whatever
, no other dir in <root>Lib\site-packages\*
should exist. Maybe others?As mentioned in conda-forge/shapely-feedstock#3, there was a patch file which wasn't being used. It would be nice if all files were inspected for their use.
Conda-smithy seems to put the built package at
/feedstock_root/build_artefacts/linux-64/tifffile-0.3.1-np19py34_0.tar.bz2
but then look to upload it from the path
dist/conda-forge/tifffile/0.3.1/linux-64/tifffile-0.3.1-np18py27_0.tar.bz2
See these excerpts from the full log.
# If you want to upload this package to anaconda.org later, type:
#
# $ anaconda upload /feedstock_root/build_artefacts/linux-64/tifffile-0.3.1-np19py34_0.tar.bz2
#
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
Using anaconda-server api site https://api.anaconda.org
[warning] Distribution linux-64/tifffile-0.3.1-np18py27_0.tar.bz2 already exists ... removing
[NotFound] The path u'/dist/conda-forge/tifffile/0.3.1/linux-64/tifffile-0.3.1-np18py27_0.tar.bz2' could not be found
Traceback (most recent call last):
File "/opt/conda/bin/anaconda", line 6, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.4/site-packages/binstar_client/scripts/cli.py", line 93, in main
description=__doc__, version=version)
File "/opt/conda/lib/python3.4/site-packages/binstar_client/scripts/cli.py", line 75, in binstar_main
return args.main(args)
File "/opt/conda/lib/python3.4/site-packages/binstar_client/commands/upload.py", line 194, in main
if remove_existing_file(binstar, args, username, package_name, version, file_attrs):
File "/opt/conda/lib/python3.4/site-packages/binstar_client/commands/upload.py", line 135, in remove_existing_file
binstar.remove_dist(username, package_name, version, file_attrs['basename'])
File "/opt/conda/lib/python3.4/site-packages/binstar_client/__init__.py", line 345, in remove_dist
self._check_response(res)
File "/opt/conda/lib/python3.4/site-packages/binstar_client/__init__.py", line 161, in _check_response
raise ErrCls(msg, res.status_code)
binstar_client.errors.NotFound: ("The path u'/dist/conda-forge/tifffile/0.3.1/linux-64/tifffile-0.3.1-np18py27_0.tar.bz2' could not be found", 404)
Traceback (most recent call last):
File "/feedstock_root/ci_support/upload_or_check_non_existence.py", line 101, in <module>
upload(cli, meta, owner, channel)
File "/feedstock_root/ci_support/upload_or_check_non_existence.py", line 51, in upload
env=os.environ)
File "/opt/conda/lib/python3.4/subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', 'binstar.token', 'upload', '--force', '/feedstock_root/build_artefacts/linux-64/tifffile-0.3.1-np18py27_0.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1 ./ci_support/run_docker_build.sh returned exit code 1
(You might notice I added the --force
argument to anaconda upload
, but I can't think this could be related to the problem.)
Is the following error message because the feedstock_content/
dir is not installed anywhere and I am running outside of the conda-smithy dir?
scopatz@athenaie ~ $ conda smithy init xonsh/recipe --feedstock-directory xonsh-feedstock
Traceback (most recent call last):
File "/home/scopatz/.local/bin/conda-smithy", line 4, in <module>
__import__('pkg_resources').run_script('conda-smithy==0.1.0.dev0', 'conda-smithy')
File "/home/scopatz/miniconda3/lib/python3.4/site-packages/setuptools-18.0.1-py3.4.egg/pkg_resources/__init__.py", line 735, in run_script
File "/home/scopatz/miniconda3/lib/python3.4/site-packages/setuptools-18.0.1-py3.4.egg/pkg_resources/__init__.py", line 1652, in run_script
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/EGG-INFO/scripts/conda-smithy", line 161, in <module>
args.subcommand_func(args)
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/EGG-INFO/scripts/conda-smithy", line 64, in __call__
generate_feedstock_content(feedstock_directory, args.recipe_directory)
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/EGG-INFO/scripts/conda-smithy", line 24, in generate_feedstock_content
configure_feedstock.main(target_directory)
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/conda_smithy/configure_feedstock.py", line 179, in main
copy_feedstock_content(forge_dir)
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/conda_smithy/configure_feedstock.py", line 77, in copy_feedstock_content
copytree(feedstock_content, forge_dir, 'README')
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/conda_smithy/configure_feedstock.py", line 61, in copytree
for item in os.listdir(src):
FileNotFoundError: [Errno 2] No such file or directory: '/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/feedstock_content'
scopatz@athenaie ~ $ ls xonsh-feedstock/
conda-forge.yml recipe
It would be neat if we could add some metadata to the recipe, and that automatically diverted the destination channel.
i.e.
extra:
channels:
- testing
And this ended up on conda-forge/channel/testing
rather than the default of conda-forge/channel/main
.
There is some detail here about whether this actually belongs in the recipe, or the conda-forge.yaml file...
When I invoke oncda smithy
to create a github repo (after setting the token correctly) I get this:
$ conda smithy github-create --user mwcraig mrjob-feedstock/
Traceback (most recent call last):
File "/Users/mcraig/miniconda/bin/conda-smithy", line 9, in <module>
load_entry_point('conda-smithy==0.1.0.dev0', 'console_scripts', 'conda-smithy')()
File "/Users/mcraig/miniconda/lib/python2.7/site-packages/conda_smithy-0.1.0.dev0-py2.7.egg/conda_smithy/conda_smithy.py", line 164, in main
args.subcommand_func(args)
File "/Users/mcraig/miniconda/lib/python2.7/site-packages/conda_smithy-0.1.0.dev0-py2.7.egg/conda_smithy/conda_smithy.py", line 94, in __call__
user_or_org.get_user(args.user)
UnboundLocalError: local variable 'user_or_org' referenced before assignment
Before I try this: would you be interested to broaden the purpose of conda-smithy to "build-smith" and make it possible to plug in different built scripts for different package managers?
Conda skeleton adds a license unknown when the license metadata is missing.
I am not sure if the linter catches that, but we should raise an error and look for a license the proper license in the repo/docs.
See https://github.com/conda-forge/geojsonio-feedstock
(The CIs registrations are OK though, they drop the .py
)
conda-forge/libtiff-feedstock#5 pinned the version of libtiff rather than being promiscuous with supported versions.
According to conda-forge policy (TBD), we can verify that specific common packages are pinned. Whilst this can't be done in general (e.g. numpy), it is rare to not want to pin things like zlib, libtiff etc.
As highlighted once upon a time by @ocefpaf, it would be good to have https://xkcd.com/1629/ in the README of conda-smithy (but explicitly not in all of the feedstocks ๐).
unknown
(see #92).BSD 3-Clause
and BSD 2-Clause
. (See source https://opensource.org/licenses/alphabetical.)When trying to get smithy help on the command line, I see the following:
scopatz@athenaie ~ $ conda smithy -h
Traceback (most recent call last):
File "/home/scopatz/.local/bin/conda-smithy", line 4, in <module>
__import__('pkg_resources').run_script('conda-smithy==0.1.0.dev0', 'conda-smithy')
File "/home/scopatz/miniconda3/lib/python3.4/site-packages/setuptools-18.0.1-py3.4.egg/pkg_resources/__init__.py", line 735, in run_script
File "/home/scopatz/miniconda3/lib/python3.4/site-packages/setuptools-18.0.1-py3.4.egg/pkg_resources/__init__.py", line 1652, in run_script
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/EGG-INFO/scripts/conda-smithy", line 10, in <module>
import conda_smithy.configure_circle_ci as configure_circle_ci
File "/home/scopatz/.local/lib/python3.4/site-packages/conda_smithy-0.1.0.dev0-py3.4.egg/conda_smithy/configure_circle_ci.py", line 13, in <module>
with open(os.path.expanduser('~/.conda-smithy/circle.token'), 'r') as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/home/scopatz/.conda-smithy/circle.token'
It would be ideal if smithy could ask for the CircleCI username and password and then put the token in the right place.
I'm currently trying to understand how to run a "feedstock" for the irkernel packages (three or four repos -> IRkernel/irkernel.github.io#16):
I don't want to lose the ability to run travis tests on github ("travis is green" -> merge the PR), but it seems that integrating the conda-smithy files in the main repos will use travis as a macosx builder and not as a testing mechanism on linux? I also don't want to have a package upload for all commits, only for some tags.
If I use a different repo (irkernel-smithy
-> much like staged-recipes), but AFAIK every time I push into that repo, a build is attempted for all packages. So my current thinking is adding a small file which only contains the repo which should be built and maybe a tag and let the other configs figure out to use that to only build that one repo and upload to a conda channel (and to our own r repo). Every time I want a tag build and pushed to the repos, I would change that file and trigger a build in that irkernel-smithy
repo.
Is this the right way or do I misunderstand the idea behind conda-smithy?
Hello All,
When testing a simple package (xz), I am seeing the following error in the circle CI build:
./ci_support/run_docker_build.sh
Containers: 0
Images: 11
Storage Driver: btrfs
Execution Driver: lxc-1.0.7
Kernel Version: 3.14.28-031428-generic
Operating System: Ubuntu precise (12.04.5 LTS) (containerized)
CPUs: 2
Total Memory: 240.2 GiB
Name: box860
ID: 6D76:EKQV:CUD4:KSVV:UGDK:OJSW:AX7Y:CIIP:HRVF:VYAP:JM65:LONE
WARNING: No memory limit support
WARNING: No swap limit support
Unable to find image 'pelson/conda64_obvious_ci:latest' locally
Pulling repository pelson/conda64_obvious_ci
FATA[0000] Error: image pelson/conda64_obvious_ci:latest not found ./ci_support/run_docker_build.sh returned exit code 1
How do you all feel about sorting all of the for loops in jinja templating? It would be nice to have these loops output in deterministic ways so that the git diffs are clearer and not filled with loops being sorted in a different order (because python 3).
That's at least what it tells me when I install it via conda.
The replacement for the binstar command in the templates seems to be anaconda auth --create --name WhateverName --scopes "repos conda api"
. Without the api
bit I couldn't get it to run :-(
If someone tries to add a package twice (e.g. to staged-recipes when it is already a feedstock), it would be nice if the linter politely told one to get rid of it and PR the changes.
Better still would be if staged-recipes could take the difference and PR them against the correct feedstock. However, that is a separate issue ( conda-forge/staged-recipes#272 ).
It would be nice if we checked if skip: True
was set for a platform as early as possible and terminate CI.
Yes:
extra:
recipe-maintainers:
- a
- b
- c
No:
extra:
recipe-maintainers:
- a
- c
- b
This may be a bit of work (so long term possibly), but it could be really useful. The linter will error our and point out something(s) wrong with the recipe in CI. I wonder if we could modify it so it actually makes these notes on the PR itself. That way it nicely integrates with the review process. As we add more functionality to the linter, I expect this will only become more valuable.
Follow on to #42.
For the scope of this ticket, the following ideas should be implemented:
To keep it concrete - any other ideas should have their own issue.
Usually it is better to use %PYTHON% setup.py install
as that prevents problems when other pythons are in the path and the ordere of the envs are wrong (not sure why this should happen but I've seen, where some kind of system python was picked up instead of the python from the conda env)...
For one project, having an (or a few) additional repo(s) for the conda builds is fine. But for things like https://twitter.com/wesmckinn/status/685537127199723523 (+answers...) I think having this layout will lead to hundreds of repos and I think that this a bad idea: you lose overview, it's a lot of work to update passwords or the smithy code itself. And getting bugs assigned to the right repo is a pain (the current ipython problem after the big split).
I would like to propose that conda-smithy gets a switch to build such a multiple recipe forge. I could do the work on this (IMO it's "just" some conditionals in the templates + small code changes in the generation to trigger these conditionals).
Would you take such a PR?
As addressed in conda-forge/pyutilib-feedstock#1:
Running command: ['python', '/home/travis/build/conda-forge/conda-forge.github.io/scripts/update_teams.py', './feedstocks_repo/feedstocks']
Traceback (most recent call last):
File "/home/travis/build/conda-forge/conda-forge.github.io/scripts/update_teams.py", line 85, in <module>
contributors = data.get('extra', {}).get('recipe-maintainers', [])
AttributeError: 'list' object has no attribute 'get'
CalledProcessError: Command '['python', '/home/travis/build/conda-forge/conda-forge.github.io/scripts/update_teams.py',
IMHO it would be nice to see the Anaconda badges for feedstocks in their Readmes. For instance, having the latest available version badge ( ). Possibly also the number of downloads badge (
).
It could also be nice to capture info about the currently supported platforms (as Windows is sometimes not supported and some things are Linux only). Though this may require some discussion with Continuum to get the API right. Added this issue ( https://github.com/Anaconda-Server/support/issues/27 ) to start discussion on this point. Broke out in separate issue ( #79 ).
Would be nice to ensure all downloads have a checksum. Could possibly also verify that it is correct.
It could also be nice to capture info about the currently supported platforms (as Windows is sometimes not supported and some things are Linux only). Though this may require some discussion with Continuum to get the API right. Added this issue ( https://github.com/Anaconda-Server/support/issues/27 ) to start discussion on this point.
There is a tool designed exactly for this for bash called shellcheck. It would be nice if we leveraged this to sniff out issues in build.sh
. While I am not currently aware of something like this for batch, it doesn't mean it doesn't exist. It would be good if we could find something similar for batch and use it to lint the batch scripts, as well.
Now that we have a linter, one of the CI services should be used to lint the recipes as they are proposed to the feedstock.
We should write a simple linter which checks the quality of our recipes.
I think I wrote this issue someplace else too, but can't find it now.
The kinds of things we want to check:
I think it would be a good idea to have a tag in each recipe's meta.yaml file that indicates where the package is maintained. the recipe travels with the built package, so this would be a way for anyone that got the package from who knows where to know where to go to report bugs, issues etc.
I understand conda now supports a "extras" section. So we could do something like:
extras:
- recipe_maintainer : https://github.com/conda-forge/udunits-feedstock
The linter will fail a recipe without a test
section even if it has run_test.*
files that will be run.
Brought up another issue and best summarized by this comment. Basically, we would want this feature so the linter commenting bot could comment on PR diffs thus providing context for users as to what is missing and where it needs to go, which is summarized this linter commenting bot issue ( conda-forge/conda-forge-webservices#3 ).
If I understood the system correctly the recipes are linted only at staged-recipes
but not as feedstocks. Does it make sense to keep running the linter in the feedstocks? (Maybe to ensure that new lint rules will be applied and that the recipe won't be changed to offend the rules?)
Hello Again,
On Circle CI, I set my BINSTAR_TOKEN
variable and I see the following error when it goes to update a package:
BUILD END: xz-5.2.2-0
# If you want to upload this package to anaconda.org later, type:
#
# $ anaconda upload /feedstock_root/build_artefacts/linux-64/xz-5.2.2-0.tar.bz2
#
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
Current conda install:
platform : linux-64
conda version : 3.18.1
conda-build version : 1.18.0
python version : 3.4.3.final.0
requests version : 2.8.0
root environment : /opt/conda (writable)
default environment : /opt/conda
envs directories : /opt/conda/envs
package cache : /opt/conda/pkgs
channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
https://repo.continuum.io/pkgs/free/noarch/
https://repo.continuum.io/pkgs/pro/linux-64/
https://repo.continuum.io/pkgs/pro/noarch/
config file : /root/.condarc
is foreign system : False
Nothing to test for: xz-5.2.2-0
Traceback (most recent call last):
File "/feedstock_root/ci_support/upload_or_check_non_existence.py", line 99, in <module>
on_channel = distribution_exists_on_channel(cli, meta, owner, channel)
File "/feedstock_root/ci_support/upload_or_check_non_existence.py", line 65, in distribution_exists_on_channel
binstar_cli.show_channel(owner=owner, channel=channel)['files']]
File "/opt/conda/lib/python3.4/site-packages/binstar_client/mixins/channels.py", line 29, in show_channel
self._check_response(res, [200])
File "/opt/conda/lib/python3.4/site-packages/binstar_client/__init__.py", line 159, in _check_response
raise ErrCls(msg, res.status_code)
binstar_client.errors.NotFound: ('The user [] does not exist', 404)
lxc-start: lxc_start.c: main: 342 The container failed to start.
lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options. ./ci_support/run_docker_build.sh returned exit code 1
The full run can be found here: https://circleci.com/gh/scopatz/xz-feedstock/7 Note that this does work without the BINSTAR_TOKEN
variable set. Any ideas?
We are running into some cases where brew is installing stuff that competes with our packages. We should start cleaning this stuff out so we don't need workarounds. For instance, brew provides its own gmp, which clashes with the one we distribute. See this thread for an example of handling this problem and this issue caused by the brew gmp. Here is some code that I wrote, which would help purge brew.
Here is a list that @ocefpaf compiled of things that brew
installed.
- apple-gcc42
- autoconf
- automake
- boost
- cgal
- cloog
- cloog-ppl015
- cloog018
- cmake
- freexl
- gcc
- gcc46
- gcc48
- gdal
- geos
- giflib
- gmp
- gmp4
- go
- gpp
- isl
- isl011
- jpeg
- json-c
- libgeotiff
- libgpg-error
- libksba
- liblwgeom
- libmpc
- libmpc08
- libpng
- libspatialite
- libtiff
- libtool
- libxml2
- libyaml
- lzlib
- maven
- mercurial
- mpfr
- mpfr2
- node
- openssl
- ossp-uuid
- pkg-config
- postgis
- postgresql
- ppl011
- proj
- pyenv
- readline
- sfcgal
- sqlite
- subversion
- wget
- xctool
Ensuring a build/number section, even if it is 0, seems to be the current practice.
Is there any way to automatically set the BINSTAR_TOKEN
environment variable on the various CI services, if we have it locally? This could be a pain to manage by hand...
I'm currently trying to build a multiple recipe version of conda-smithy and found that all access token go into a subdir of ~
, but only the BINSTAR_TOKEN
is requested to be in os.environ
. Is it ok, to change that to also use a file based access token ( ~\.conda-smithy\anaconda.token
)?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.