researchapps / sherlock Goto Github PK

repository for collaborating with sherlock users to create containers

License: MIT License

C 39.09% Shell 13.44% Dockerfile 44.97% Stan 2.49%

hpc sherlock containers singularity cluster scientific-computing

sherlock's Introduction

Sherlock

This is a repository for container build files to help users of Sherlock. If you are a user and need help, please submit an issue and we will help you build a container! When you are happy with your container, we recommend that you add the Singularity or Dockerfile file to a new repo, and build automatically with Singularity Hub or Docker Hub. Generally, your workflow will look like the following:

Install Singularity and/or Docker to work locally
Ask for help via an issue if you don't know how to start
Create a build specification file, a text file called Singularity, for your software needs. You can start with another user's as an example. You can also start with a Dockerfile, and then pull it to a Singularity image from Docker Hub.
Ask for help with your file! This is what this repo is here for. You can submit issues with questions, and we will discuss and work together on the issues.
Test your build locally.

Installation

You should first install Singularity and Docker so that you can build images on your host. If you use a Mac, you will need to install Singularity in a virtual machine like Vagrant. Singularity is going to allow us to interact exactly the same, but with an image that we can use on Sherlock! The biggest difference is that a Singularity image is a read online, single file (a format called squasfs so it is compressed) that we can physically move around and execute like a script.

General Usage

On your local machine, building usually looks something like the following:

  sudo singularity build mynewimage.img Singularity

If it has a runscript, you can run as follows:

  singularity run mynewimage.img # or
  ./mynewimage.img

If you are having trouble with the runscript, shell inside like this to look around. The runscript is a file at the base of the image (/) called singularity.

 singularity shell mynewimage.img

You can also (on your local machine) use the --writable option to test installation of software. You should have your build file open in another window and copy down commands that work, and ensure that the entire build goes successfully from start to finish without an error. Remember, any command that you issue and don't write done is NOT reproducible!

Singularity Hub

Note to Sherlock users: this functionality is not yet added to Sherlock, but will be available upon the next release of Singularity in March. For now, you can upload images the old school way (FTP). When this is enabled, you will be able to push the build file to Github, and then link your repo to Singularity Hub. Then using the image on sherlock will come down to:

  module load system
  module load singularity/2.4
  singularity run shub://reponame/mynewimage

sherlock's People

Contributors

Stargazers

Watchers

Forkers

dbdean

sherlock's Issues

Using the python3 in container for setting up jupyter kernel

Hello,

I am trying to use the python3 installed in the container (which python gives opt/conda/bin/python) to show up as a kernel in jupyter.

To do so, in the location ${HOME}/.local/share/jupyter/kernels, I specify a new directory called mykernel and added a kernel.json file of the following description:

{
 "language": "python",
 "argv": ["home/lalit/singularity",
   "exec",
   "-w",
   "/home/lalit/singularity/test.simg",
   "/opt/conda/bin/python",
   "-m",
   "ipykernel",
   "-f",
   "{connection_file}"
 ],
 "display_name": "Python 3 (Singularity)"
}

But this gives a kernel not found error (shown below) upon starting a jupyter notebook. I think the paths to the singularity and test.simg are correct and this error stems from some permissions issue. Any clue on how to solve this? Thanks.

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
    return self.result()  # May raise too.
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/handlers.py", line 72, in post
    type=mtype))
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/manager.py", line 259, in start_kernel
    **kw)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/manager.py", line 204, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/launcher.py", line 138, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied

VEP not exposed

From discussion in an email thread, moved here for documentation and preservation:

So it seems that pvacseq may work using:

singularity exec --bind $PWD:/scif/data/pvacseq --app pvacseq pvacseq.simg pvacseq -h

I will test that with real data ASAP.

I thought that similarly VEP should work with

singularity exec --bind $PWD:/scif/data/VEP --app VEP pvacseq.simg VEP
/.singularity.d/actions/exec: 9: exec: VEP: not found

singularity run --bind $PWD:/scif/data/VEP --app VEP pvacseq.simg
No Singularity runscript for contained app: VEP

Dockerfile for pytorch in python2 won't build

Hi, thanks for the help in getting pytorch running in python 2.7 using your Dockerfile at https://github.com/researchapps/sherlock/blob/master/pytorch-dev/Dockerfile.py2. I did have to make some changes to get it to work however:

+++ Dockerfile.new      2018-08-14 09:44:08.542990180 +1000
@@ -20,17 +20,19 @@
      /opt/conda/bin/conda config --add channels conda-forge && \
      /opt/conda/bin/conda install -y numpy pyyaml scipy ipython mkl && \
      /opt/conda/bin/conda install -y matplotlib scikit-learn scikit-image && \
+     /opt/conda/bin/conda install -y typing && \
      /opt/conda/bin/conda install -c intel mkl-include && \
      /opt/conda/bin/conda install -c pytorch magma-cuda90 && \
      /opt/conda/bin/conda clean -ya
-RUN /opt/conda/bin/conda update -y base conda
+RUN /opt/conda/bin/conda update -y conda
 ENV PATH /opt/conda/bin:$PATH
 # This must be done before pip so that requirements.txt is available
 RUN git clone https://github.com/pytorch/pytorch /opt/pytorch && \   
     cd /opt/pytorch && \
     git submodule update --init
 RUN pip install --upgrade pip
-RUN TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1 7.0+PTX" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
+RUN cd /opt/pytorch && \
+    TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1 7.0+PTX" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
     CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
     pip install -v .
 RUN git clone https://github.com/pytorch/vision.git && cd vision && pip install -v .

I hope this is useful if this problem is reproducible for other people.

researchapps / sherlock Goto Github PK

sherlock's Introduction

Sherlock

Installation

General Usage

Singularity Hub

sherlock's People

Contributors

Stargazers

Watchers

Forkers

sherlock's Issues

Using the python3 in container for setting up jupyter kernel

VEP not exposed

Dockerfile for pytorch in python2 won't build

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent