galaxyproject / starforge Goto Github PK

View Code? Open in Web Editor NEW

8.0 25.0 16.0 3.29 MB

Build Galaxy dependencies, wheels, and other things, in Docker (and other things)

Home Page: https://galaxyproject.org/

License: MIT License

Shell 28.91% Python 62.18% Makefile 7.50% Dockerfile 1.41%

usegalaxy dependencies wheels python

starforge's Introduction

DEPRECATED

This project is now deprecated in favour of wheelforge, which uses PyPA projects (like cibuildwheel) to build wheels for Python packages.

Starforge: Build Galaxy things in virtualization

Things you can do with Starforge:

Build Galaxy Tool Shed dependencies
Build Python Wheels (e.g. for the Galaxy Wheels Server)
Rebuild Debian or Ubuntu source packages (for modifications)

These things will be built in Docker. Additionally, wheels can be built in QEMU/KVM virtualized systems.

Documentation can be found at starforge.readthedocs.org

Starforge is maintained by the Galaxy Project and community. A list of contributors to the project can be found on GitHub.

Quick Start

For all types of builds, begin by installing Docker.

Tool Shed Dependencies

There are two scripts that can be used, depending on the package recipes available:

$ ./build.sh <package>
$ python build.py <package> --version 1.0

build.sh is the older format, and simply uses a single <package>build.sh file, like Atlas. build.py is the newer format, and uses yaml metadata in <package>/<version>/build.yml.

The base image for Galaxy packages is Debian Squeeze. This will hopefully produce binaries usable on Galaxy's targeted platforms (at time of writing: CentOS 6+, Debian 6.0+, Ubuntu 12.04+).

$ ./build galaxy <package>
$ python build.py <package>

To build packages against a different OS, you can use the --image flag, e.g.:

$ ./build <dist>[:tag] <package>
$ python build.py <package> --image <dist>[:tag]

e.g.

$ ./build ubuntu:trusty nginx
$ python build.py nginx --image debian:squeeze

Building all the things:

There's a separate build-all.sh which allows you to build all of the packages using their preferred build mechanism

Notes on the two build scripts

build.py

The <version> option is optional, and defaults to the string 'default', which is useful for recipes that don't have version specific changes (E.g. bcftools 1.0 builds identically to 1.2)

Python Wheels

Starforge can build both pure Python and C-extension Python modules into wheels for Linux under Docker and for Mac OS X under QEMU/KVM. To do this, you'll want to install Starforge (preferably in a Python virtualenv) using pip install starforge (to install from PyPI) or python setup.py install to install from the source.

Docker (and QEMU) images to use are specified in starforge/config/default.yml. To modify this file, copy it to $XDG_CONFIG_HOME/galaxy-starforge/config.yml ($XDG_CONFIG_HOME is, by default ~/.config). The sample file wheels/build/wheels.yml is used to determine what wheels can be built and their rules for building. To use this file, use the --wheels-config argument to starforge wheel or copy wheels.yml to $XDG_CONFIG_HOME/galaxy-starforge/wheels.yml.

Wheels can be built using starforge wheel <package>, e.g.:

$ starforge wheel pycrypto
$ starforge wheel --no-qemu pysam   # only build on docker

See the output of starforge --help for help using the Starforge command-line interface.

Pull Request wheel builder

Pull requests to the Starforge repository on Github that modify wheels/build/wheels.yml can automatically be built for all specified platforms on a dedicated Starforge build server by the Galaxy Jenkins service. To utilize, modify wheels.yml as appropriate and create a pull request. Any member of the Galaxy Committers group can then authorize Jenkins to initiate the build. If it fails, you can modify the pull request and further builds can be triggered.

Notes on images

Linux

Images used to build wheels are uploaded to the Starforge Docker Hub repo and will be pulled as necessary. Typically you will only use the manylinux1-wheel and manylinux1-32-wheel images, which are manylinux CentOS 5-based images that will usually produce wheels usable on all Galaxy-supported platforms.

You can also produce "platform-specific" wheels by using the full-wheel imageset. This is useful if you want to link to distribution-specific system versions of non-standard libraries without bundling them in to the wheel.

Mac OS X

Mac OS X images are not provided due to legal reasons. Consult the :doc:`osx` documentation for details.

starforge's People

Contributors

Stargazers

Watchers

Forkers

bgruening davebx docker-a natefoo alex-docker nsoranzo chambm kidaak mvdbeek jmchilton martenson jxtx nturaga abretaud pcm32 nuwang

starforge's Issues

Can't build sdists if the imageset doesn't contain a docker image

Traceback (most recent call last):
  File "/home/jenkins/sfvenv/bin/starforge", line 11, in <module>
    sys.exit(starforge())
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/commands/cmd_wheel.py", line 108, in cli
    itervalues(wheel_config.images))[0]
IndexError: list index out of range

And in the case of Jenkins, this fails the entire build. Building sdists are of limited use anyway, since they are trivial to build manually and some packages do not create working sdists due to invalid manifests. I've temporarily disabled building sdists by commenting it out of /home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/commands/cmd_wheel.py.

Convert from shell to YAML-based build system

Do others think this is a good idea? Should we do away with the shell script builds entirely?

btrfs subvol delete needs to wait for QEMU shutdown

Otherwise you can get:

Running: ssh root@fauxsx -- shutdown -h now
Shutdown NOW!

System shutdown time has arrived
Deleting snapshot: btrfs subvolume delete /home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de
Transaction commit: none (default)
Delete subvolume '/home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de'
ERROR: cannot delete '/home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de' - Device or resource busy
Traceback (most recent call last):
  File "./wheel.py", line 328, in <module>
    main()
  File "./wheel.py", line 324, in main
    osx_qemu(args)
  File "./wheel.py", line 238, in osx_qemu
    subprocess.check_call(delete_cmd)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['btrfs', 'subvolume', 'delete', '/home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de']' returned non-zero exit status 1

Wait a few more seconds and it's fine:

(starforge)nate@mjolnir0% btrfs subvolume delete /home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de
Transaction commit: none (default)
Delete subvolume '/home/nate/snapshots/@75a0e152-8337-11e5-9891-002436f4a7de'

Namespace cached files

Currently if I fetch a master.tar.gz from Github this would be saved in Starforge's cache as master.tar.gz and reused for any other attempt to fetch a master.tar.gz. Either some sort of randomized/hashed naming should be used, or at least have subdirectories of the cache directory namespaced to the package name.

Do not use unversioned changeable tarballs

If I link to a tarball of a branch of a project on Github, the URL is static but the contents of the tarball are not. These should not be used in build.yml (but currently are).

please rebuild nginx and upload to launchpad

After the recent glibc CVE and updates, the nginx package fails to run, complaining about config that requires the upload module. Can you or someone affiliated with the project roll a new nginx package for distribution? The docker-galaxy container also relies on this version of nginx, so it's failing like so:

unknown directive "upload_store" in /etc/nginx/nginx.conf:117

Should we start versioning packages?

E.g. we have an emboss folder which is specifically 6.6.0? Thoughts? @bgruening @natefoo

Install pip in /python/cp*

This is already the case for manylinux (it is part of the manylinux base image) but not the OS X, purepy, and platform-specific images.

Samtools.

Samtools needs a couple extra binaries: galaxyproject/tools-iuc#146

PR which fixed the manually built package: galaxyproject/tools-iuc#155

Wheel build images are not in Docker Hub

And now that we have a Galaxy account, we should probably move them (and the other images) under that account.

Splitting build into different layers

installing dependencies and downloading tarballs can take some time. It would be nice to have this as extra Docker layers so we do not need to repeat this during testing.

apt-get extra layer
download extra layer
--no-cache switch for ./build to overwrite the layers

OS X building fails without __platform_cache.yml

A remnant from "full platform" wheels (dropped in #117), when building OS X/macOS wheels, Starforge attempts to collect the platform string using the Galaxy wheel fork's get_platforms() before attempting to build wheels, unless you have a ~/.local/share/galaxy-starforge/__platform_cache.yml with at least:

{'starforge/osx-wheel:10.6': macosx_10_6_intel}

Otherwise, you get this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'get_platforms'

Followed by this:

Transaction commit: none (default)
Delete subvolume '/home/dannon/snapshots/@aafdd210-949c-11e7-9113-002436f4a7de'
Traceback (most recent call last):
  File "/home/dannon/.venv/bin/starforge", line 11, in <module>
    sys.exit(starforge())
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/starforge/commands/cmd_wheel.py", line 88, in cli
    for name in forge.get_expected_names():
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/starforge/forge/wheels.py", line 75, in get_expected_names
    self.image.plat_specific)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/starforge/cache.py", line 189, in platform_cache
    plat_specific=plat_specific)
  File "/home/dannon/.venv/local/lib/python2.7/site-packages/starforge/cache.py", line 108, in cache
    ).splitlines()[0].strip()
IndexError: list index out of range

Use `pip download` to cache sdists

Now that pip 8 is out, we can use pip download instead of the convoluted pip install-but-don't-install method currently in use to download sdists for caching.

Create per-job venvs for Jenkins

Jenkins currently uses /home/jenkins/sfvenv to run starforge. Instead, it should just create a virtualenv, pip install starforge, and run starforge from there

builds aren't tarred into single directory

I think this is a problem which affects most of our build recipes. The tars contain bin/, lib/, etc at the root level, rather than being packaged in a much nicer $pkg_name-$version-$arch/{bin,lib}/ (or similar).

The fact that we don't have "nice" archives means that statements like this are needed: https://github.com/galaxyproject/tools-iuc/pull/179/files#diff-99e82d7ee179c109e1940be4c8994711R8

Need R packages

xref galaxyproject/tools-iuc#101

Problem with python-ldap and ldaps://

The current python-ldap wheel doesn't work with secured connection to ldap server (ldaps://)
Galaxy issue: galaxyproject/galaxy#3178

The error is:

Traceback (most recent call last):
File "/home/galaxy/galaxy-dist/lib/galaxy/auth/providers/ldap_ad.py", line 126, in authenticate
l.simple_bind_s()
File "/home/galaxy/galaxy-dist/.venv/local/lib/python2.7/site-packages/ldap/ldapobject.py", line 223, in simple_bind_s
resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
File "/home/galaxy/galaxy-dist/.venv/local/lib/python2.7/site-packages/ldap/ldapobject.py", line 514, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/home/galaxy/galaxy-dist/.venv/local/lib/python2.7/site-packages/ldap/ldapobject.py", line 521, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/home/galaxy/galaxy-dist/.venv/local/lib/python2.7/site-packages/ldap/ldapobject.py", line 106, in _ldap_call
result = func(*args,**kwargs)
SERVER_DOWN: {'desc': "Can't contact LDAP server"}

Debugging this is not easy... If I find some time I'll try to setup a small docker test system and see what's wrong.

`starforge wheel` needs an option to fail if any expected wheels are missing after attempting to build

Otherwise, the Jenkins build cannot fail when wheels fail to build. First noticed in #44.

Automate release process

Current process:

Update history, documentation, version
Tag, push tags
Update wheels.yml for new Starforge version, PR, test wheel/sdist, merge
twine upload -s starforge-<version>-*
ssh jenkins@mjolnir0 ./sfvenv/bin/pip install --upgrade starforge <-- Jenkins could just do this on every build
Update image configs to install the new Starforge version
Rebuild all of the Docker/QEMU images =/
Push the Docker images to Docker Hub.

`brew install` fails due to non-writable root dirs

When using sudo -u admin brew install as root, $HOME remains ~root, causing failures due to a non-writable home directory:

Error: Permission denied - /var/root/Library
Warning: Bottle installation failed: building from source.
Error: Permission denied - /var/root/Library
Traceback (most recent call last):
  File "/python/wheelenv/bin/starforge", line 9, in <module>
    load_entry_point('starforge==0.3', 'console_scripts', 'starforge')()
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/python/wheelenv/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/python/wheelenv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/python/wheelenv/lib/python2.7/site-packages/starforge/commands/cmd_bdist_wheel.py", line 67, in cli
    forge.bdist_wheel(output=output, uid=uid, gid=gid)
  File "/python/wheelenv/lib/python2.7/site-packages/starforge/forge/wheels.py", line 193, in bdist_wheel
    'brew', 'install'] + pkgs, cwd='/tmp')
  File "/python/wheelenv/lib/python2.7/site-packages/starforge/forge/wheels.py", line 103, in execute
    run(cmd, cwd=cwd)
  File "/python/wheelenv/lib/python2.7/site-packages/starforge/execution/local.py", line 28, in run
    check_call(cmd, cwd=cwd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-u', 'admin', 'brew', 'install', 'xz']' returned non-zero exit status 1

The fix needs to be applied to the starforge inside the image.

manylinux1 and platform-specific capabilities conflict

We have a hack in place that allows us to install Starforge in our manylinux1 docker images because Starforge currently depends on a fork of wheel to build platform-specific wheels (which is unwanted for manylinux1 wheels).

Since we probably don't need to build platform-specific wheels going forward (but maybe should not drop support for them entirely), Starforge should support these concurrently. The changes in the wheel fork can be spun into a separate library to accomplish this.

The working dir for bdist_wheel is the last source dir, not the first

As discovered by @abretaud here, the working directory for python setup.py bdist_wheel is supposed to be the directory of the first source tarball (e.g. the one downloaded from PyPI) but instead, it's the last one, due to overwriting the root variable in the loop here.

I am not too worried about breaking existing recipes by fixing this, though, as we can just fix any broken recipes when those wheels are upgraded.

PyYAML 3.11 wheel is not Python3-compatible

I'm trying to run Galaxy unit tests in a Python3 virtualenv and this is the first thing that broke:

Collecting PyYAML==3.11 (from -r requirements.txt (line 4))
  Downloading https://wheels.galaxyproject.org/packages/PyYAML-3.11.tar.gz (138kB)
    100% |████████████████████████████████| 143kB 1.2MB/s 
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/PyYAML.egg-info
    writing top-level names to pip-egg-info/PyYAML.egg-info/top_level.txt
    writing pip-egg-info/PyYAML.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/PyYAML.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/PyYAML.egg-info/SOURCES.txt'
    error: package directory 'lib3/yaml' does not exist

The reason seems to be that https://wheels.galaxyproject.org/packages/PyYAML-3.11.tar.gz is missing files and dirs with respect to https://pypi.python.org/packages/75/5e/b84feba55e20f8da46ead76f14a3943c8cb722d40360702b2365b91dec00/PyYAML-3.11.tar.gz#md5=f50e08ef0fe55178479d3a618efe21db

Maybe while fixing this we can also move to the latest version 3.12 .

Use the holy-build-box as underlying container

@natefoo what do you think about: https://github.com/phusion/holy-build-box

Babel 2.4 doesn't build with starforge.

Low priority issue - but if we ever want to start checking updates to our packages this will be a problem. See commits in #128.

package_bowtie_2_2_4 Executable Permissions - can't run bowtie as non-galaxy user

Installed bowtie2 from the toolshed, which brings in package_bowtie_2_2_4, which downloads bowtie binaries from depot.galaxyproject.org

http://depot.galaxyproject.org/package/linux/x86_64/bowtie2/bowtie2-2.2.4-linux-x86_64.tgz

Jobs submitted on our setup, running as a ‘real user’ (not galaxy user) via pulsar + SLURM fail with a permissions error. This is due to the permissions on some of the bowtie executables in the .tar.gz from depot being owner/group executable only. Other users cannot run bowtie2:

-rwxrwx---. 1 galaxy galaxy 18K Oct 22 2014 bowtie2

If I download the tar.gz from depot manually and extract then I get:

-rwxrwx--- 1 dtrudgian biohpc_admin 18K Oct 22 2014 bowtie2

Could the .tar.gz be fixed, so that the executables will work in a run as real user configuration? Is there any sensible way Galaxy could enforce permissions to guard against this issue?

A Galactic Name

Since this is sorta useful, should we give it a Galactic name? Since it's about building things, the first things that popped into my head were:

Stardust
Nebula
Starforge
??

A Galactic Logo

@erasche? =D

Built data should be pushed to depot

Just an issue for tracking purposes.

Jenkins build reports success when actually failed.

Looking at build 90

Detecting changes to wheels.yml...
Traceback (most recent call last):
  File "/home/jenkins/sfvenv/bin/starforge", line 11, in <module>
    sys.exit(starforge())
  <SNIP>
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/config/wheels.py", line 55, in open
    return cls(global_config, config_file=config_file)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/config/wheels.py", line 62, in __init__
    self.load_config()
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/config/wheels.py", line 75, in load_config
    purepy=False)
  File "/home/jenkins/sfvenv/local/lib/python2.7/site-packages/starforge/config/wheels.py", line 36, in __init__
    self.imageset = imagesets[config.get('imageset', default_imageset)]
KeyError: 'default_plus_libpython-wheel'
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Setting status of b15a158cfe9dc68600b011a1109f7136da516c08 to SUCCESS with url https://jenkins.galaxyproject.org/job/starforge-wheels/90/ and message: 'Build finished. '
Using context: wheel-build
Finished: SUCCESS

we can see an obvious failure. wheels/build/jenkins.sh probably needs a set -ex

Generate missing wheels

I don't think we have wheels for most of these - I could be wrong though...

python-genomespaceclient==0.1.8
  - boto3 [required: Any, installed: 1.4.7]
    - botocore [required: <1.8.0,>=1.7.0, installed: 1.7.35]
      - docutils [required: >=0.10, installed: 0.12]
      - jmespath [required: <1.0.0,>=0.7.1, installed: 0.9.3]
      - python-dateutil [required: <3.0.0,>=2.1, installed: 2.6.1]
        - six [required: >=1.5, installed: 1.11.0]
    - jmespath [required: <1.0.0,>=0.7.1, installed: 0.9.3]
    - s3transfer [required: <0.2.0,>=0.1.10, installed: 0.1.11]
      - botocore [required: <2.0.0,>=1.3.0, installed: 1.7.35]
        - docutils [required: >=0.10, installed: 0.12]
        - jmespath [required: <1.0.0,>=0.7.1, installed: 0.9.3]
        - python-dateutil [required: <3.0.0,>=2.1, installed: 2.6.1]
          - six [required: >=1.5, installed: 1.11.0]
      - futures [required: <4.0.0,>=2.2.0, installed: 3.1.1]
  - python-swiftclient [required: Any, installed: 3.4.0]
    - futures [required: >=3.0, installed: 3.1.1]
    - requests [required: >=1.1, installed: 2.10.0]
    - six [required: >=1.5.2, installed: 1.11.0]
  - requests [required: Any, installed: 2.10.0]

Fix mercurial and pysam sources on wheels.galaxyproject.org

The sdists for both mercurial 3.5 and 3.7.3 and pysam 0.8.* are invalid and should be replaced. Mercurial with the upstream tarballs and pysam with the source tarballs in Github specified in wheels.yml, with the corresponding prebuild changes applied.

Should wheels/build/wheels.yml live in Galaxy?

As pointed out by @chambm, updating this file is really a Galaxy-specific task, not Starforge. Should it live in the Galaxy repo instead?

One downside: The Jenkins job that checks for changes to wheels.yml and attempts to build the wheels would need to run on Galaxy instead, and would run for every Galaxy PR despite only being applicable to a very small percentage of them.

Print a list of the contents of dist/ after building wheels

Sometimes the built package has a different filename than Starforge expects, it'd be very useful to know what distutils et. al. actually built when this happens.

Incorporate build.py in to the Starforge CLI

I wrote the CLI for the wheel work I was doing, but it should be general enough to use with the other build types.

sdist command packages the setup.py wrapper

For packages that use insert_setuptools, starforge sdist causes the sdist to contain the setup.py wrapper rather than the distribution's setup.py. This resulted in some invalid tarballs on wheels.galaxyproject.org (see #180).

Rather than mess with files, we might be able to just:

$ python -c 'import setuptools; __file__ = "setup.py"; exec(compile(open("setup.py").read(), "setup.py", "exec"))' bdist_wheel

I tested this by hand with the upstream Markdown 2.6.3 sdist and it works. More complex setup scripts might have issues.

Once working, there would probably be no harm in running all sdist and wheel builds this way by default. This would be easily done by changing the implicit default of insert_setuptools from false to true, and running a traditional python setup.py bdist_wheel if insert_setuptools is explicitly set to false for a particular wheel.

Alias docker images internally

Some of the images used by Starforge are simply extra tags on other images:

starforge/purepy-wheel:latest ➡️ starforge/debian-wheel:6
starforge/base-wheel:latest ➡️ starforge/debian-wheel:6
starforge/base32-wheel:latest ➡️ starforge/debian32-wheel:6

This results in the "aliases" appearing as separate images in Docker Hub, which is annoying/confusing. In addition, each "image" requires a separate push (which doesn't make sense to me, if they have all the same underlying layers and the same image ID shouldn't pushing aliases of already-pushed images result in "Image already exists" messages for each layer? Unfortunately that's not the case:

nate@weyerbacher% docker push starforge/debian-wheel:6
The push refers to a repository [starforge/debian-wheel] (len: 1)
8a40c04093cd: Image already exists 
ee2d5171de71: Image successfully pushed 
2e3c8128fa02: Image successfully pushed 
5b7a9e06dfda: Image successfully pushed 
991a3085cc57: Image successfully pushed 
d0e6975c75d2: Image successfully pushed 
60eb7aaceb8a: Image successfully pushed 
11f17e6c581c: Image successfully pushed 
134ff8959711: Image successfully pushed 
134ff8959711: Buffering to Disk 
b7336fb73419: Image already exists 
9e94b6f5e953: Image already exists 
d834be925c79: Image already exists 
4c115374adfc: Image successfully pushed 
Digest: sha256:a406dcc8d6b33cecfbe6bad12123527ec10dae61067782709419c45525d0cfe6
nate@weyerbacher% docker push starforge/purepy-wheel:latest                                                                                                                             ~/git/starforge/wheels/image
The push refers to a repository [starforge/purepy-wheel] (len: 1)
8a40c04093cd: Image already exists 
ee2d5171de71: Image successfully pushed 
ee2d5171de71: Buffering to Disk 
5b7a9e06dfda: Image successfully pushed 
991a3085cc57: Image successfully pushed 
d0e6975c75d2: Image successfully pushed 
d0e6975c75d2: Buffering to Disk 
11f17e6c581c: Image successfully pushed 
134ff8959711: Image successfully pushed 
134ff8959711: Buffering to Disk 
b7336fb73419: Image already exists 
9e94b6f5e953: Image already exists 
d834be925c79: Image already exists 
4c115374adfc: Image successfully pushed 
Digest: sha256:78d097eaded004cfa31252007533834beb6dd5b5d2d74f08b5b6a4574b79122d
nate@weyerbacher% docker push starforge/base-wheel:latest                                                                                                                               ~/git/starforge/wheels/image
The push refers to a repository [starforge/base-wheel] (len: 1)
8a40c04093cd: Image already exists 
ee2d5171de71: Image successfully pushed 
ee2d5171de71: Buffering to Disk 
5b7a9e06dfda: Image successfully pushed 
991a3085cc57: Image successfully pushed 
d0e6975c75d2: Image successfully pushed 
d0e6975c75d2: Buffering to Disk 
11f17e6c581c: Image successfully pushed 
134ff8959711: Image successfully pushed 
134ff8959711: Buffering to Disk 
b7336fb73419: Image already exists 
9e94b6f5e953: Image already exists 
d834be925c79: Image already exists 
4c115374adfc: Image successfully pushed 
Digest: sha256:b4d14a1f197898b0b78d401207d8b200e2a833b1e9070fb9e5a5ab7813fc2f22

Need Perl Packages

It sure would be nice to have gross statistics on which packages take the longest to compile/install...since that's a bit of a UX issue, if we knew which ones needed pre-compilation, we could do that.

Wrong wheels for python-ldap

Hi!
This is a followup to #97 (it will be more visible than a closed PR 😄 )
The python-ldap wheels that were uploaded to wheels.galaxyproject.org are not the good ones.They come from https://depot.galaxyproject.org/starforge/wheels/build-132/ while the good ones are in https://depot.galaxyproject.org/starforge/wheels/build-133/
The only difference between the 2 builds is a49f517 that fix a linking problem.

Getting UCSC tools into the depot

@natefoo can we get the UCSC binaries into the depot? We need them for a data manger and some converters.

build_all.sh is out of date.

Can someone update that please?

Add watchdog required dependency for Galaxy

Currently it is optional, but it would be nice if it came automatically. There are some features I would enable by default. It is a bit tricky probably because it has different dependencies on different platforms.

https://pypi.python.org/pypi/watchdog

Starforge official Hymn

https://www.youtube.com/watch?v=sHDmXtW9Yx0&list=RDsHDmXtW9Yx0&index=1

(this is what you listen to when compiling, right?)

Some prebuild actions should run when building sdists

9008c57 made it so that the prebuild action would only run when building wheels, but this behavior is incorrect when building sdists where the prebuild step would modify the version number of the built package. Thus, it's necessary to have separate prebuild actions for building wheels, building sdists, and building both.

Handle wheels that are purepy on some platforms and platform-specific on others

For example, watchdog is pure python on Linux but platform on macOS.

We could in theory add it to both sections, and restrict the platform entry to just the macos-wheel image set. This won't work currently because the purepy/platform dicts are collapsed to a single OrderedDict in the config code.

We could also try to collapse the purepy/python dicts to one and just add a purepy key to the wheel entries (defaults to true) that can either take a boolean or a hash with items <key>: <bool> where <key> is an imageset, image, or platform? Not sure which makes the most sense. Probably imageset.

% starforge wheel --no-docker --wheels-config=wheels/build/wheels.yml dictobj
Traceback (most recent call last):
  File "/home/john/workspace/starforge/.venv/bin/starforge", line 9, in <module>
    load_entry_point('starforge==0.0.1.dev0', 'console_scripts', 'starforge')()
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/click-6.2-py2.7.egg/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/john/workspace/starforge/.venv/lib/python2.7/site-packages/starforge-0.0.1.dev0-py2.7.egg/starforge/commands/cmd_wheel.py", line 108, in cli
    for name in forge.get_sdist_expected_names():
UnboundLocalError: local variable 'forge' referenced before assignment

Select a random port for forwarding SSH when running under QEMU

And create an ephemeral ssh config directory and ssh config. This is roughly what Vagrant does.