Code Monkey home page Code Monkey logo

idle's Introduction

Integrated Deep Learning Engine (IDLE)

IDLE provides nice cross-compiler for * TensorFlow and * Caffe with simple format, called UDLI (Unified Deep Learning Interface).
Overall architecture of IDLE is introduced in * Taewoo Kim, Eunju Yang, Soyoon Bae and Chan-Hyun Youn, “IDLE: Integrated Deep Learning Engine with Adaptive Task Scheduling on Heterogeneous GPUs”, to appear in IEEE TENCON, 2018

Getting Started

Dependencies

  • Ubuntu 16.04 (>=)
  • python 2.7
  • TensorFlow-gpu >= 1.90
  • Caffe 1.0
  • cuDNN-6.0
  • CUDA 9.0

Install prerequisite : caffe, pyCaffe, TensorFlow

Caffe

Python 2.7

$ sudo apt-get install python-pip python-dev 

Liberaries.

$ sudo apt-get update
$ sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
$ sudo apt-get install --no-install-recommends libboost-all-dev

Install CUDA : * CUDA 9.0

BLAS.

$ sudo apt-get install libatlas-base-dev

Other dependencies

$ sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

Install Caffe

// Download source code
$ cd {workspace}
$ git clone https://github.com/BVLC/caffe.git
$ cd caffe
$ cp Makefile.config.example Makefile.config
// Change Make configuration --> check python include, lib path
$ vi Makefile.config
# NOTE: this is required only if you will compile the python interface. # We need to be able to find Python.h and numpy/arrayobject.h. PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/local/lib/python2.7/dist-packages/numpy/core/include/

# We need to be able to find libpythonX.X.so or .dylib. PYTHON_LIB := /usr/lib
// Compile
$ make clean
$ make all
$ make test
$ make runtest

Install * PyCaffe

  • Dependencies
$ cd {workspace}/caffe/python
$ for req in $(cat requirements.txt); do sudo pip install $req; done
  • Setting environment variables
$ echo export PYTHONPATH={caffe_path}/python:$PYTHONPATH >> ~/.bashrc
$ echo export PYTHONPATH={python_path}:$PYTHONPATH >> ~/.bashrc
$ echo export CAFFE_ROOT={caffe_path} >> ~/.bashrc
$ echo export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH >> ~/.bashrc $ source ~/.bashrc
  • Recompiile
$ make clean
$ make pycaffe
  • Test PyCaffe
$ python
>> import caffe
TensorFlow
$ sudo apt-get install python-pip python-dev
$ pip install tensorflow-gpu
Python dependency module

Parmiko

$ pip install parmiko

Pillow

$ pip install pillow
Setting SSH Keys (for distributed training)

To support distributed training, all workers should share ssh-key each other.

  • SSH key generation (local machine)
$ ssh-keygen -t -rsa
$ ls -al ~/.ssh
  • send each ssh-key to all workers
$ scp ~/.ssh/id_rsa.pub (remote_machine_id@ip_address):($destination)
  • register each worker key to authorized_keys
$ cat $destination/id_rsa.pub >> ~/.ssh/authorized_keys

Install IDLE

Install IDLE on your environment.

$ cd {working directory}
$ git clone https://github.com/KAIST-NCL/IDLE.git
$ echo export PYTHONPATH={working directory}:$PYTHONPATH >> ~/.bashrc
$ source !$

How to Run

Sample DL-MDL files are wtored under /model folder. all files with dlmdl extension shows the examples.

Edit sample DLMDL file

Edit the sample files for your environment.

Running IDLE
$ cd {working directory}
$ python src/UDLI.py --framework=tf --input=model/VGG -r

arguments are described in below

Argument Explanation Example
--framework Choose framework --framework = tf , --framework=caffe
--input Path of input file (.dlmdl file) --input=model/VGG.dlmdl
-r Execution or not (If -r is set, execution is conducted right after compilation -r
--compile_out compile output path --compile_out=output/VGG
--log_out path to save TensorFlow log --log_out=/tmp/IDLE_LOG/TF

License

This project is released under the *BSD 2-Clause license, see LICENSE for more information.

idle's People

Contributors

liebhaber avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.