Get Singularity image built and running on NSG/XSEDE resources about neuronunit HOT 7 CLOSED

@rgerkin

My complete workflow (including preparing singularity images for execution, and invoking them on NSG-HPC) is documented in the README.md I will keep updating that file as I make more debugging progress.

Plans, manually upload image data pnp_latest-2017-05-19-f1d9712ba440.img to Comet using web portal, launch job from webportal.

Invoke the singularity in bootstrap mode using a configuration script such as:
nu.def

sudo singularity bootstrap pnp_latest-2017-05-19-f1d9712ba440.img nu.def

See if I can configure the launch of singularity+Comet to basically execute that statement, using pnp_latest-2017-05-19-f1d9712ba440.img nu.def, which I will manually upload to the portal.

from neuronunit.

bump. ^^

from neuronunit.

After today's discussion about the simpler pathway to from docker 2 singularity, I was motivated to follow through with it. However I noticed that using the pattern:
http://singularity.lbl.gov/docs-docker

sudo singularity create --size 4000 tensorflow.img
sudo singularity import tensorflow.img docker://tensorflow/tensorflow:latest

But substituting in relevant variable names to our neuronunit build context:

sudo singularity create --size 4000 pnp.img
sudo singularity import pnp.img docker://pnp

Causes the error below (with singularity on native ubuntu):

Error getting tags using url https://registry-1.docker.io/v2/library/pnp/tags/list
Bootstrap initialization
No bootstrap definition passed, updating container
Executing Prebootstrap module
Executing Postbootstrap module
ERROR: Container does not contain the valid minimum requirement of /bin/sh

This has been fixed in the development branch of singularity:
apptainer/singularity#421
Which probably involves building it from source, which involves its own technical obstacles.

This is the proposed solution (only requires a docker installation), singularity is not even needed its very verbose but its all in one line:

https://github.com/singularityware/docker2singularity

from neuronunit.

On my Mac I had this problem (long irrelevant code blocks replaced by ...):

vagrant@vagrant-ubuntu-trusty-64:~$ sudo singularity create --size 10000 neuronunit.img
Creating a new image with a maximum size of 10000MiB...
...
Done.
vagrant@vagrant-ubuntu-trusty-64:~$ sudo singularity import neuronunit.img 
...
Extracting /home/vagrant/.singularity/docker/sha256:7719a427e59f2abb32c0722f1bfb670a6c4a7339c0e8d2fffc19a8eb3ae2d606.tar.gz
tar: opt/conda/lib/icu/current: Cannot create symlink to ‘56.1’: File exists
tar: Exiting with failure status due to previous errors
ERROR:python:Error extracting image: /home/vagrant/.singularity/docker/sha256:7719a427e59f2abb32c0722f1bfb670a6c4a7339c0e8d2fffc19a8eb3ae2d606.tar.gz
ERROR: Aborting with RETVAL=1

from neuronunit.

@russelljjarvis
But it looks like Singularity 2.3 has now been released (as of May 31st) so if I rename my old singularity-vm directory, and then do:

mkdir singularity-vm
cd singularity-vm
vagrant init singularityware/singularity-2.3
vagrant up
vagrant ssh

i.e. Option 1 on the Mac instructions page

then inside the Vagrant container do:
singularity --version (should be 2.3 with some suffix)
then do:

singularity create --size 10000 neuronunit.img
singularity import neuronunit.img docker://scidash/neuronunit:latest

It seems to complete without error. Then I do:
singularity shell neuronunit.img

and

python -c "import neuronunit"

and it seems to work.

from neuronunit.

AOK.

from neuronunit.

Probably not useful in light of so many HPC resources providing Docker support.

from neuronunit.