lablup / backend.ai-kernels Goto Github PK

This is an issue delegating lablup/sorna-agent#1.
(It is too long to type in the first line of commit messages...)

Some languages offer standardized logging (e.g., Python's logging module and Julia's info(), warn() functions). Let's wrap them and provide a prettier output by distinguishing them via a separate type of stream: "log". (Currently the new PUSH/PULL agent protocol only recognizes "stdout", "stderr", "media", "finished", "waiting-input" message types.)

Many deep learning codes requires a lot of memory and computation time.
We need some automated way to measure the maximum memory used and computation time for a given example code, for better capacity planning and scheduler designs.

• Update python3 related kernel to python 3.6

• Basic REPL support
• Pre-install a list of essential packages
  • qqplot
  • more?

Add C++ language support

ref) https://github.com/NVIDIA/nvidia-docker

We need to rebuild our caffe/tensorflow images based on NVIDIA's cuda base images.

Add support for Rust language

• Basic REPL implementation
• Pre-install a list of essential packages

• Haskell support
• For language specific course.

• Python 3 (this is the reference implementation)
• Python 3 - TensorFlow
• Python 3 - TensorFlow GPU
• Python 3 - Caffe
• Python 2
• PHP
• R
• Julia
• Javascript
• Lua
• Haskell
• Octave
• Git shell (command part)

For Python kernels, we also need to update sorna-media package to v0.3.

Minimum features:

• Binds PUSH/PULL ZeroMQ sockets on TCP port 2000 and 2001
• Console outputs should be sent as multipart messages in this format: [b'stdout', b'utf8-encoded-text'] and [b'stderr', b'utf8-encoded-text'].
• No more separate exception handler. Just print language-native traceback to stderr.
• Send a multipart message [b'finished', b''] when execution is done.

Optional features:

• Strip tracebacks to include only user-provided codes.
• Provide interactive input function using input socket (like self.handle_input in Python 3 impl.)
  • If possible, override the most commonly used, standard input function.

Tips:

• Copy and modify test_run.py in python3 kernel directory to test the main programs before building docker containers for fast iteration of debugging and development.

Add a custom Git command shell for Git tutorial courses.

TensorFlow v1.0 is released last week.

Egoing has reported an issue that he could not see the result of the following code:

var 입력한비밀번호 = '1111';
var 소금의크기 = 32;
var 암호화반복횟수 = 10000;
var 암호의길이 = 32;
var crypto = require('crypto');
crypto.randomBytes(소금의크기, function(오류, 소금){
    crypto.pbkdf2(입력한비밀번호, 소금, 암호화반복횟수, 암호의길이, 'sha512', function(오류, 생성된암호){
        console.log(생성된암호.toString('hex'));
    });
});

This is due to the current nodejs kernel just goes through the synchronous part until it sends the execution result and callbacks generated by the user code are executed later.
We need a "blocking" mechanism until all user callbacks finish as well as temporarily removing existing sorna-side callbacks from the event loop.

As the result, I have found a small hacky open source project that uses C++ addon to access uv_run() function, and patched it to implement a blocking call until all callbacks finish:
abbr/deasync#53

Then, I have added unref() / ref() support to zeromq.node project:
JustinTulloss/zeromq.node#503

Now we can implement a proper blocking call for nodejs4 kernel.

Add Go language support.

• Basic REPL for Golang
• Pre-install a list of essential packages
  • availability of go get-like functionality?

Many kernels reuse the same intial Dockerfile procedures. Let's enable caching for them.

A collection of custom package inclusion requests.

Several code examples (e.g., this one) using TensorFlow crash due to thread limits imposed by our jail.

terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted

The root cause is libeigen (a C++ matrix calculation library) used by TensorFlow which reads OMP_NUM_THREADS environment variable to initialize its thread pool.

ref) http://eigen.tuxfamily.org/dox/TopicMultiThreading.html

• Basic Lua execution support
• Useful packages?

Jail should be compiled in Linux (preferably the same version of Ubuntu as REPL kernels use) and thus native Docker environments requires a separate Ubuntu image setup.
Let's add some helper scripts for building new jail binaries.

Add support for PyTorch.

• Shell
  • Uncomment set convert-meta off in /etc/inputrc to allow output of 8-bit characters
  • Run locale-gen en_US.UTF-8 and set LANG environment variable so that bash can handle multi-byte UTF-8 characters correctly. (e.g., backspace should delete each Unicode char like a single char.)
• Vim
  • Add terminal encoding and indentations to /etc/vim/vimrc.local
  • Note: /etc/vim/vimrc and /usr/share/vim/vim74/debian.vim already has syntax highlighting, eol, nocompatible settings.

More to come.

• Basic REPL implementation
  • ref) https://github.com/esromneb/zeromq-octave
• Pre-install a list of essential packages

... so that other people can easily update the service images.
Currently, we use only a single sorna instance, sorna.lablup, but this should be extended to cover multiple instances via docker-registry.lablup.

In development phase, many engineers often interrupt ongoing executions when they realize something is going to be wrong. Jupyter notebook also supports interrupts using SIGINT signal from the notebook server to the kernel process. Let's support it.

There are some issue to think:

• SIGINT signal may not be caught if it is delivered in the middle of a system call since Python 3.5+ retries interrupted I/O syscalls. ref: PEP-475
• Abruptive interruption may leave the process under inconsistent state. (e.g., somewhere between sending ZeroMQ multi-part output messages)

initialization of multiarray raised unreported exception

• TensorBoard – replaced by lablup/cloud.backend.ai#38 (internal issue)
• [ ] [TensorFlow Fold](https://github.com/tensorflow/fold → Not maintained.
• https://github.com/microsoft/tensorwatch
• more?

Interactive terminal support for tutorial/workshops.

TensorFlow kernels are not working on high-end servers due to process/thread limits in our jail.
This is probably due to mis-reporting of sysconf(_SC_NPROCESSORS_ONLN) library call, which reports the full CPU count instead of Docker-allocated cpuset.

Phase 1: Allow some fixed hosts for lectures and tutorials, such as:

• Allow connections to DNS servers (TCP/UDP port 53)
  • probably we could cache DNS resolution results to restrict connections to hosts
• Allow connections to the following hosts:
  • github.com, bitbucket.org (TCP 22, 80, 443): source-code repository services
  • httpbin.org (TCP 80, 443): HTTP protocol test server
  • example.com, example.org, example.net, and possibly other IANA-managed sample domains (TCP 80, 443): example domain site reserved by IANA

Phase 2: Allow customization

• Allow additional host/port pairs specified per client-side session ID via Sorna API.

Keras is a wrapper around existing DL libraries.
Let's add support for it as two separate kernels images: tf + keras / theano + keras.
tf + keras image will be an upgrade of the current python3-tensorflow images.

sqlite-based data manipluation course
(demands exist at research/consulting firms, even without programming skills)

• Basic REPL implementation
• Pre-install a list of essential packages

/home/joongi/venv-ipython/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Remove or suppress above warning messages when a fresh kernel first uses matplotlib.
Maybe we could run font cache building process during docker builds.

• Python 3.5 container

Add C language support

For PL classes.

Add Java language support

Some C/C++ libraries used by kernel's 3rd-party packages implicitly spawn as many threads as the number of available CPU cores, and this exceeds the default child process/thread limit (32) in servers with a high number of cores. It causes crashes or indefinite hangs of kernels. 😞

• Limit the maximum number of CPU cores per container (default is 1 and customizable by each kernel via image labels)
• (in lablup/sorna-agent) Implement a CPU core allocation policy that takes NUMA nodes and core occupancy of existing kernels into account.

Write a set of parametrized test suites that uses language/version-specific example codes to test basic zero-mq REPL functionality of new/updates images.

Some users have tried input() in Python kernels during code-golf sessions at conferences. In such case they saw "unexpected" timeouts because most request-reply based kernels cannot handle user inputs.

Until we have a nice user-input handling in the front-ends, we need to explicitly disable them and show error messages to the user.

On AWS p2.xlarge instance, TF kernels give the following warnings:

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.

My initial tests show that it's viable to migrate to Alpine Linux for our kernel images.

• sorna-jail works well if compiled inside Alpine Linux. (Binaries compiled in Ubuntu has glibc-specific symbol references: __fprintf_chk, __vfprintf_chk)
• The image size of a Python 3.6 base kernel (without numpy/scipy stuffs but including jail support) is 112 MB only! (previously it was ~600 MB)

The core ideas to reduce image size are:

• Alpine Linux tiself. The base image is less than 8 MB, just containing a busybox, the package manager apk, and a few utililties such as scanelf. libc is replaced with musl transparently.
• Remove build-only depenencies at the end of a single RUN command so that each image layer has small footprints.
• The package manager (apk) in Alpine Linux provides a concept of "virtual" package installation space, so we can easily purge a set of packages. Also, most Alpine Linux pacakges are made to be independent with minimal cross-dependencies.

Challenges remainig:

• Migrate CUDA kernels to base our Alpine images. I think this is theorectically possible, but there may be unexpected glibc-specific dependency in CUDA binaries.

apt-get has a cli argument --no-install-recommends that skips installation of recommended packages but the main dependencies. This could reduce the kernel image sizes a lot.
Let's test this.

NOTE: Since we have basic unit tests for docker images, it would be sufficient to check if the tests are passed after building images after applying --no-install-recommends.

• Basic REPL implementation
• Pre-install a list of essential packages
  • Some ZEND packages?

Docker now supports multi-staged builds in the stable releases. Let's use it.

• Support configuration for each REPL container
• Examples
  • Memory Limit
  • Execution time limit
  • It should also be supported by ingen requests

Let's support execution of shell scripts as well in the query mode.

• Prevent killing the REPL daemon by intercepting kill() system call to a specific pid.
• Prevent access to the REPL code (/home/sorna)

Proxy issue for lablup/sorna-agent#30.