CUDNN_STATUS_INTERNAL_ERROR while running main.py about surfacenet HOT 4 OPEN

@Rubikplayer
For the updated error_log, it mentions: pygpu.gpuarray.GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory. Can you change the cnmem=0.75 --> cnmem=0.95 in .theanorc OR change __GPUMemoryGB = 11 to a safe value, say __GPUMemoryGB = 6 in params.py and let's see what it print out.

Also, for the theano installation please refer to #3 (comment)

from surfacenet.

@mjiUST
The code seems to be running, after I set gpuarray.preallocate=0.8 (also commented #cnmem=0.75). (This was before I saw your feedback. I will try your suggested values a bit later).

May I confirm with you on two questions:

• Theano/Lasagne is quite new to me. I wasn't quite sure the difference between gpuarray.preallocate and cnmem.

According to the theano doc link, seems gpuarray.preallocate was designed for new gpu back, and cnmem for the old one. Since we are using version 0.9, I suppose I should set cnmem instead of gpuarray.preallocate? If so, then what I just set was just not setting any limit.

• With my setting above, it seems to run on the example dinosaur data. About 2 hours passed, it finished 68% in surfacenet inference. Is this typical, or there's any way to make it faster?

My setting change: __GPUMemoryGB = 11 and __cube_D = 32.
Also, my GPU (1080 Ti) should be slower than Titan X.

Thanks for the help!!

from surfacenet.

@Rubikplayer
Thanks for your feedback. It's great to know the code is running.

• For the theano memory preallocation, the link you mentioned says that after you set the Theano flag allow_gc to False (Theano will not collect GPU memory garbage.), CNMeM will not affect GPU speed anymore. In my opinion, CNMeM and gpuarray.preallocate are the same thing for older and newer versions. Just use any one which let the GPU memory preallocated in the very beginning (you can use command watch nvidia-smi to check, i.e., the majority memory was reserved.)

• For the speed of SurfaceNet: the setting __cube_D = 64 could result in a little bit faster process. Before that you can check whether your .theanorc include optimizer=fast_run for fast running mode as mentioned in

  SurfaceNet/installEnv.sh

  Line 40 in 149f6e0

  echo -e "[global] \nfloatX=float32 \ndevice=cuda \noptimizer=fast_run \n \nallow_gc=True \ngpuarray.preallocate=-1 \n \nnvcc.fastmath=True \n" > ~/.theanorc

  
  If everything goes well, the dinosaur dataset should finish in one hour.

from surfacenet.

@mjiUST
Thanks for the suggestion! I tried optimizer=fast_run indeed accelerates the process. but for __cube_D = 64, I still got some out of memory issue. I've sent an email to your school email for detail questions.

Thanks again!

from surfacenet.