cuda Runtime Error (77): an illegal memory access was encountered about ssd.pytorch HOT 7 CLOSED

If used CUDA_VISIBLE_DEVICES to assign just a single GPU, comment
net = torch.nn.DataParallel(ssd_net)
It works for me.

from ssd.pytorch.

Try using

os.environ['CUDA_LAUNCH_BLOCKING'] = 1

as :

Sometimes the error message can be caught at a different place compared to where it is occuring because of the asynchronous nature of CUDA (pasted from there)

from ssd.pytorch.

@JoachimHugonot Okay! I will use this in future runs. Though this error is quite arbitrary.

from ssd.pytorch.

@chauhan-utk x
Could you try to scale the loss in :

from ssd.pytorch.

Hi! What is your slight modification? I have trained the model recently using batch size 32 through 120k iterations without any issues, so I cannot comment without knowledge of the change you made.

from ssd.pytorch.

I had changed the line 88 in multibox_loss.py to include a custom loss function. So far my observation have been that loc_p contains the best single prediction for the corresponding ground truth loc_t and so we can use something other than smooth L1 loss. The resultant saved in loss_t is a single dimension tensor just as it would by using the existing loss function. Rest everything was same.

Since this is not a particular issue with the existing code and the above methods have been somewhat effective, I am closing this issue.

from ssd.pytorch.

I had similar issues. Any thoughts?

from ssd.pytorch.