Different workers always process the same batch about cloudml-samples HOT 6 CLOSED

Thanks for the response, I just wanted to be sure this is how it was working, I am making my own image network based on this code and I want all workers to pitch in for one epoch because the processing is very heavy (3D images). I am using a shared queue with filenames (each file is an image) instead of examples precisely to address the issue you have mentioned.

from cloudml-samples.

@MiguelMonteiro if you find yourself performance constrained, you can also use a shared queue for the filenames, and a local queue for the shuffle_batcher, and split up your epoch between num_worker files, then only filenames will traverse the network, but workers will all work on different examples.

from cloudml-samples.

Hahah, Didn't see your earlier comment. Looks like you are already doing exactly what I suggested =).

from cloudml-samples.

My understanding is there's a performance trade-off to be had here (which doesn't really matter much as the census dataset is so small, but from a pedagogical perspective...). If shared_name is specified input data is must traverse the network several additional times: read as binary to the worker -> enqueued to variable on a param-server -> read as binary to worker -> enqueued as parsed tensor to param server -> read as parsed tensor to worker

Making the memory (for buffering) and network requirements of the cluster much larger. Now in this case, a single census example is quite small so you are probably right, we should be using global variables for the queues, but for networks with large features (like image networks), this would normally be the right approach.

from cloudml-samples.

@puneith I think for the move into census we should use shared_queue=True most likely.

from cloudml-samples.

@MiguelMonteiro This code uses tf.datasets in TF1.4 now. Please see https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset#shard and use that appropriately in the code. Feel free to send a PR as well as I see this as a generic sample feature.

from cloudml-samples.