how long does NMT model run on GPU's? about unsupervisedmt HOT 4 CLOSED

Hi,

On 1 GPU it takes about 4 or 5 days to converge for En-Fr. But after 2 or 3 days the model is usually not very far from the final performance. It is not so straightforward to make the code work on multi-GPU, because the back-translation process (which for now generates back-parallel data on CPU threads) may slow down the training a lot. In practice, the best for a multi-GPU setting would be to have some GPU generating back-parallel data, while some others perform the training, but we have not implemented this yet.

from unsupervisedmt.

The more CPU the better. Using 20 CPU generates back-parallel data twice faster than with 10 CPU, and prevents the GPU from waiting too long for incoming data to train on. With 40 CPU, the GPU is almost never waiting, so this is the ideal scenario.

from unsupervisedmt.

@glample Thanks for those insights. It helps.
How about timelines for different multi CPU's (1,4,16,32 etc) ?

from unsupervisedmt.

Great Thanks @glample

from unsupervisedmt.