Code Monkey home page Code Monkey logo

deepwater's Introduction

Deep Water

What it is

  • Native implementation of Deep Learning models for GPU-optimized backends (MXNet, Caffe, TensorFlow, etc.)
  • State-of-the-art Deep Learning models trained from the H2O Platform
  • Train user-defined or pre-defined deeplearning models for image/text/H2OFrame classification from Flow, R, Python, Java, Scala or REST API
  • Behaves just like any other H2O model (Flow, cross-validation, early stopping, hyper-parameter search, etc.)
  • The next best thing after sliced bread
  • Under development

What it is not

  • An oil drilling platform

Python/R Jupyter Notebooks

Check out a sample of cool Deep Learning Jupyter notebooks!

Pre-Release Downloads

This latest release of Deep Water is based on H2O-3 3.10.3.1 "Tverberg-1" 1b263c79 - Jan 28 2017

The downloadable packages below are built for the following system specifications:

  • Ubuntu 16.04 LTS
  • Latest NVIDIA Display driver
  • CUDA 8.0.44 or later (we recommend the latest version) in /usr/local/cuda
  • CUDNN 5.1 or later (placed inside of lib and include directories in /usr/local/cuda/)

To use the GPU, please set the following environment variables:

export CUDA_PATH=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_PATH/lib64:$LD_LIRBARY_PATH

Python + Flow (most common)

R + Flow (R users)

Flow (Web UI)

If you are interested in running H2O Deep Water on a different infrastructure, see the DIY build instructions below.

Running GPU enabled Deep Water in H2O

(Optional) Launch H2O by hand and build Deep Water models from Flow (localhost:54321)

java -jar h2o.jar

Java example use cases

Example Java GPU-enabled unit tests.

Python example use cases

Example Python GPU-enabled unit tests. Check out a sample of cool Deep Learning Python Jupyter notebooks!

R example use cases

Example R GPU-enabled unit tests. Check out a sample of cool Deep Learning R Jupyter notebooks!

Scala / Sparkling Water example use cases

Coming soon.

Pre-Release Amazon AWS Image

For your convenience, here's a pre-built image for Amazon's EC2 environment, based off our recent H2O Open Tour Hands-On Deep Water workshop. (Recording coming soon.)

  • AMI ID: ami-10bd9607
  • AMI Name: deepwater-dallas-v4
  • Recommended instance types: p2.xlarge
  • After launching the instance, you can connect to port 8888 (Jupyter Notebook) or port 54321 (H2O Flow).

Refer to the Deep Water Workshop on EC2 Amazon AMI document for additional information on how to run this AMI.

Pre-Release Docker Image

We have a GPU-enabled Docker image on Docker Hub. To use it you need a Linux machine with at least one GPU, and with docker and nvidia-docker installed.

An NVIDIA GPU with a Compute Capability of at least 3.5 is necessary. See https://developer.nvidia.com/cuda-gpus .

If you use Amazon Web Services (AWS), a good machine type to use is the P2 series. Note that G2 series machines have GPUs that are too old.

  1. Install Docker, see http://www.docker.com

    • Optional Step. Make docker run without sudo. Instructions for Ubuntu 16.04:
      • sudo groupadd docker
      • sudo gpasswd -a ${USER} docker
      • sudo service docker restart
      • log out then log in, or newgrp docker
  2. Install nvidia-docker, see https://github.com/NVIDIA/nvidia-docker . Note that you can only use Linux machines with one or more NVIDIA GPUs:

    • GNU/Linux x86_64 with kernel version > 3.10
    • Docker >= 1.9 (official docker-engine, docker-ce or docker-ee only)
    • NVIDIA GPU with Architecture > Fermi (2.1) and Compute Capability >= 3.5
    • NVIDIA drivers >= 340.29 with binary nvidia-modprobe
  3. Download and run the H2O Docker image

    • nvidia-docker run -it --net host -v $PWD:/host opsh2oai/h2o-deepwater
    • You now get a prompt in the image: # . The directory you started from is avaiable as /host
    • Start H2O with java -jar /opt/h2o.jar
    • Python, R and Jupyter Notebooks are available
    • exit or ctrl-d closes the image

Roadmap, Architecture and Demo

Download the Deep Water overview slides.

architecture architecture architecture architecture

DIY Build Instructions

If you want to use Deep Water in H2O-3, you'll need to have a .jar file that includes backend support for at least one of MXNet, Caffe or TensorFlow.

1. Build MXNet

Instructions to build MXNet

2. Build TensorFlow

Instructions to build TensorFlow

3. Build Caffe

Coming soon.

4. Build H2O Backend Connectors

From the top-level of the deepwater repository, do

./gradlew build -x test

This will create the following file: build/libs/deepwater-all.jar

5. Add DeepWater support to H2O-3

You need to check out the h2o-3. Copy the freshly created jar file build/libs/deepwater-all.jar from the previous step to H2O-3's library h2o-3/lib/deepwater-all.jar (create the directory if it's not there) and you're done!

Build H2O-3 as usual:
./gradlew build -x test

This H2O version will now have GPU Deep Learning support!

To use the GPU, please make sure to set your path to your CUDA installation:

export CUDA_PATH=/usr/local/cuda
Install the Python wheel:
sudo pip install h2o-3/h2o-py/dist/h2o-3.11.0.99999-py2.py3-none-any.whl
(Optional) Install the MXNet Python/R packages

If you want to build your own MXNet models from Python or R, install the MXNet wheel (which was built together with MXNet above):

sudo easy_install deepwater/thirdparty/mxnet/python/dist/mxnet-0.7.0-py2.7.egg
R CMD INSTALL deepwater/thirdparty/mxnet/mxnet_0.7.tar.gz

Releasing

The release process bundles all defined submodules and push them into Maven central via Sonatype repository provider. The released artifacts are Java 6 compatible.

The release can be invoked for all modules by:

./gradlew -PdoRelease -PbuildOnlyBackendApi -PdoJava6Bytecode=true -Prelease.useAutomaticVersion=true release

The process performs the following steps:

  • Updates gradle.properties and removes SNAPSHOT and increases minor version (can be changed)
  • Creates a new release commit and tags it with release tag. (See gradle/release.gradle file to override the default template.)
  • Builds
  • Verifies compatibility of used API with Java 6 API
  • Bytecode rewrite to be compatible with Java 6
  • Generation of artifact metadata
  • Pushes artifacts into staging area at https://oss.sonatype.org/

The process needs to be finished manually by:

Note: The release process creates two new commits and a new tag with the release version. However, the process does not push it to a remote repository, so it is necessary to perform a remote update manually using git push --tags or update the gradle/release.gradle settings and remove the --dry-run option from the pushOptions field.

deepwater's People

Contributors

arnocandel avatar avkash avatar ledell avatar mdymczyk avatar mistobaan avatar mmalohlava avatar mstensmo avatar surekhajadhwani avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.