RuntimeError: CUDA error: an illegal memory access was encountered about tabnet HOT 5 CLOSED

This looks like a setup problem with your GPU, are you able to run other models using your GPU?

Or maybe try to reduce your batch size.

It does not seem directly related to tabnet but to pytorch/your gpu.

from tabnet.

I have never seem this problem either... Maybe something related to the CUDA version/driver version? I'm running on CUDA 10.1.

Perhaps this can help: https://docs.nvidia.com/deploy/cuda-compatibility/index.html.

from tabnet.

@eduardocarvp I experienced the same issue , your gpu is running out of memory , restart the process and it will work fine .It worked for me

from tabnet.

It worked after restarted my kernel indeed. I have this issue because of the large size of my dataset (about 25GB). What I'm still unable to do right now is to run my model multiple times as I'm trying different hyper parameters without restarting my kernel each time. I'm on a google cloud instance and it takes about 20 min just to read this large dataset. Basically I would need to find a way to free up gpu memory from previous models as if I was fitting for the first time every time. Have tried gc.collect() but didn't work so far.

from tabnet.

@amartin211 you could have a look here : https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530
torch.cuda.empty_cache() would be more effevtive to clean the GPU I guess

from tabnet.