Hey Arimo contributors,
Thanks for open-sourcing your TensorSpark!
I have done few modifications to a fork of your repo that are mostly relevant to someone interested in taking TensorSpark to production in yarn-cluster mode with CPU machines. You can go through my commit comments to see if there is anything you’d like to bring to your repo; feel free to let me know and I’ll send a PR for the branch and you can then “git cherry-pick” the commits you’re interested in.
I’ve been working with the MNIST dataset; if I find the time, I’ll try to create the ImageNet/AlexNet scenario which I see to be on your roadmap as well.
Thanks.

in file "paramservermodel.py"
int function def train(self, labels, features):

for i in range(len(self.compute_gradients)):

    #     self.gradients[
    #         i] += self.compute_gradients[i][0].eval(feed_dict=feed)       

because its not necessary to use "for" in this code, replaced by:

grads, test_error_rate = self.session.run([self.compute_gradients,self.error_rate],feed_dict=feed)
self.gradients[:] = [g[0] for g in grads]

this will save several times of gpu train, especial in mnist one iter time from 7ms to 1ms in my computer

Hi,

I have been able to run this code on an HPC cluster. Currently, I'm trying to figure out how to test with this set up.

Any suggestion would be useful,
Thank you!

After I have trained the model, I can not find an interface to save the trained model to HDFS. Is there any way that could solve this problem? much thanks.

for example : I had change the files to load my own pic like [None,32,32,3] . Everything is OK, but when I set the partition=2 or 4 , 8 ... and my computer information is gtx1070, ubuntu14.04, 8G. I also change the model init code with:
config = tf.ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.allocator_type = 'BFC' #config.gpu_options.per_process_gpu_memory_fraction = 0.2 session = tf.Session(config=config)
upon will enable several process in one gpu.
the bug is when the programer run some epoches , I find "nvidia-smi" 's gpu memory grows without stop.
from 800MB to 2G , 4G, 8G... finally show some errors like cuda OOM.
my way to solve it:
after my check and try to fix it, I find a function leads to the GPU Memory Leak
def reset_gradients(self): #with self.session.as_default(): #self.gradients = [tf.zeros(g[1].get_shape()).eval() for g in self.compute_gradients] self.gradients = [0.0]*len(self.compute_gradients) # my modify self.num_gradients = 0
though I don't the details why this change can works ,but it did.
email :[email protected]