Really cool project that I tried to recreate. Unfortunately, after testing, the discrete_vae.py gives the following error because of on inplace operation I cannot find:
Model is now CUDA!
[TRAIN - 0] GS: 1, Loss: 1.19789: 0%|โ | 1/202 [00:06<21:26, 6.40s/it]:: Entering Testing Mode
[TEST - 0]: 0%| | 0/1 [00:02<?, ?it/s]
:::: Loss: 1.2889413833618164 | 0/1 [00:01<?, ?it/s]
[TRAIN - 0] GS: 2, Loss: 1.2198: 1%|โโ | 2/202 [00:10<16:54, 5.07s/it][W ..\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in SoftmaxBackward. Traceback of forward call that caused the error:
trainer.train(
_, loss, _=model(d)
result = self.forward(*input, **kwargs)
return self.module(*inputs[0], **kwargs[0])
result = self.forward(*input, **kwargs)
softmax = F.softmax(encoding, dim = 1)
ret = input.softmax(dim)
(function _print_stack)
[TRAIN - 0] GS: 2, Loss: 1.2198: 1%|โโ | 2/202 [00:11<18:45, 5.63s/it]
Traceback (most recent call last):
trainer.train(
loss.backward()
torch.autograd.backward(self, gradient, retain_graph, create_graph)
Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 3000, 16, 16]], which is output 0 of SoftmaxBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Could you give me a hint, where the inplace operation might be? Running this code on windows on python 3.8.7 64-bit if it helps!