Cuda OOM when using ddp about svoice HOT 3 CLOSED

Thanks, it's training now with ddp after setting L 16

from svoice.

Hi @Durgesh92,
What do you mean after the 1st epoch? Do you mean that the model is being trained and suddenly you get an OOM error? What is the sampling rate of your data?
You can try to set L: 16, it should help.
You can also decrease R or decrease segment, however, these will probably hurt the overall model performance.

from svoice.

My data is 16Khz, When using ddp I get OOM after these

[2020-12-24 23:29:11,386][svoice.solver][INFO] - Train | Epoch 1 | 49960/249804 | 1.5 it/sec | Loss 17.33249
[2020-12-25 08:34:59,062][svoice.solver][INFO] - Train | Epoch 1 | 99920/249804 | 1.5 it/sec | Loss 17.00609
[2020-12-25 17:41:22,327][svoice.solver][INFO] - Train | Epoch 1 | 149880/249804 | 1.5 it/sec | Loss 17.05395
[2020-12-26 02:47:45,908][svoice.solver][INFO] - Train | Epoch 1 | 199840/249804 | 1.5 it/sec | Loss 17.07950

And when DDP is not enabled I get

[W python_anomaly_mode.cpp:60] Warning: Error detected in Log10Backward. Traceback of forward call that caused the error:
[2020-12-26 04:09:55,132][main][ERROR] - Some error happened
Traceback (most recent call last):
File "train.py", line 120, in main
_main(args)
File "train.py", line 114, in _main
run(args)
File "train.py", line 95, in run
solver.train()
File "/home/sysadmin/svoice/svoice/solver.py", line 122, in train
train_loss = self._run_one_epoch(epoch)
File "/home/sysadmin/svoice/svoice/solver.py", line 219, in _run_one_epoch
loss.backward()
File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 185, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 127, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: Function 'Log10Backward' returned nan values in its 0th output.

Looks like something is wrong with backward loss?

Thanks for the options though, I will try changing L

from svoice.