Using DDL for distributed processing on GPU
Simple MNIST Classifier using shallow neural network - ddl.py
MNIST recognition using CNN - deep_mnist.py
ddl = tf.load_op_library('/opt/DL/ddl-tensorflow/lib/ddl_MDR.so')
Initializing DDL with number of proccess and working directives with the CPU with tf.Session(config=config) as sess:
with tf.device('/cpu:0'):
rank, size, gpuid = sess.run(ddl.init(4, mode = '-mode r:4 -dump_iter 100'))
Using the gpu, start the graph definition inside this scope with tf.device('/gpu:%d' %gpuid):
Loading the Matrix Variables into the GPU's memory with ddl.bcast W = tf.Variable(ddl.bcast(tf.zeros([784, 10])))
b = tf.Variable(ddl.bcast(tf.zeros([10])))
After computing the gradients with Tensorflow's optmizer executing an all_reduce operation for the average and applying the result to the network opt = tf.train.GradientDescentOptimizer(0.5)
grads_and_vars = opt.compute_gradients(cross_entropy)
grads, vars = zip(*grads_and_vars)
grads_and_vars_ddl = zip(ddl.all_reduce_n(grads, op='avg'), vars)
objective = opt.apply_gradients(grads_and_vars_ddl)
Training the data in mini-batches dividing it in each GPU with tf.Session(config=config) as sess:
sess.run(tf.global_variables_initializer())
for i in range(1000):
batch_x, batch_y = mnist.train.next_batch(batch_size*size)
batch_x = np.split(batch_x,size)[rank]
batch_y = np.split(batch_y,size)[rank]
sess.run(objective, feed_dict={x: batch_x, y_: batch_y})
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent