Insufficient GPU Memory despite 7gbs unallocated about fastsurfer HOT 7 CLOSED

[INFO: run_prediction.py: 121]: Running view aggregation on cuda
[INFO: run_prediction.py: 263]: The memory requirements exceeds the available GPU memory, try using a smaller batch size (--batch_size ) and/or view aggregation on the cpu (--viewagg_device 'cpu').Note: View Aggregation on the GPU is particularly memory-hungry at approx. 5 GB for standard 256x256x256 images.

Have you tried setting this to cpu, i.e. --viewagg_device cpu?

from fastsurfer.

Hi, thank you for the reply

I have tried with --viewagg_device cpu - with 16gbs of ram, 2gbs swap
./run_fastsurfer.sh --t1 /home/pavgreen/Documents/LRGS/MRI/freetest/9024/mprage/anat/9024.nii --sid 9024 --sd /home/pavgreen/Documents/LRGS/MRI/freetest/PS2 --viewagg_device cpu --parallel

Though I end up with a different error

python3.8 /home/pavgreen/fastsurfer/FastSurferCNN/run_prediction.py --t1 /home/pavgreen/Documents/LRGS/MRI/freetest/9024/mprage/anat/9024.nii --aparc_aseg_segfile /home/pavgreen/Documents/LRGS/MRI/freetest/PS2/9024/mri/aparc.DKTatlas+aseg.deep.mgz --conformed_name /home/pavgreen/Documents/LRGS/MRI/freetest/PS2/9024/mri/orig.mgz --sid 9024 --seg_log /home/pavgreen/Documents/LRGS/MRI/freetest/PS2/9024/scripts/deep-seg.log --vox_size min --batch_size 1 --viewagg_device cpu --device auto
[INFO: run_prediction.py: 341]: Checking or downloading default checkpoints ...
[INFO: run_prediction.py: 158]: Output will be stored in: /home/pavgreen/Documents/LRGS/MRI/freetest/PS2
[INFO: misc.py: 159]: Using device: cuda
[INFO: run_prediction.py: 121]: Running view aggregation on cpu
[INFO: inference.py: 95]: Loading checkpoint /home/pavgreen/fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_coronal_v2.0.0.pkl
[INFO: inference.py: 95]: Loading checkpoint /home/pavgreen/fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_sagittal_v2.0.0.pkl
[INFO: inference.py: 95]: Loading checkpoint /home/pavgreen/fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_axial_v2.0.0.pkl
[INFO: run_prediction.py: 359]: Analyzing single subject /home/pavgreen/Documents/LRGS/MRI/freetest/9024/mprage/anat/9024.nii
[INFO: run_prediction.py: 247]: Successfully saved image as /home/pavgreen/Documents/LRGS/MRI/freetest/PS2/9024/mri/orig/001.mgz
[INFO: run_prediction.py: 176]: Conforming image
Input: min: 0 max: 8055
rescale: min: 0.0 max: 4180.545 scale: 0.06099683175279778
Output: min: 0.0 max: 255.0
[INFO: run_prediction.py: 247]: Successfully saved image as /home/pavgreen/Documents/LRGS/MRI/freetest/PS2/9024/mri/orig.mgz
Traceback (most recent call last):
File "/home/pavgreen/fastsurfer/FastSurferCNN/run_prediction.py", line 407, in
raise e
File "/home/pavgreen/fastsurfer/FastSurferCNN/run_prediction.py", line 376, in
pred_data = eval.get_prediction(orig_fn, data_array, orig_img.header.get_zooms())
File "/home/pavgreen/fastsurfer/FastSurferCNN/run_prediction.py", line 206, in get_prediction
pred_prob = torch.zeros(shape, **kwargs)
RuntimeError: [enforce fail at alloc_cpu.cpp:66] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 21206401024 bytes. Error code 12 (Cannot allocate memory)
ERROR: Segmentation failed QC checks.

from fastsurfer.

This indicates that you do not have enough memory on the CPU either. What resolution does your image have? In the section "System Requirements" in our README (https://github.com/Deep-MI/FastSurfer#system-requirements) you can get an idea how much memory is needed for which resolution. If you run out of memory with a 7 GB GPU and 16 GB CPU RAM, I assume your image is far below 0.7 mm. Is that correct?

from fastsurfer.

Thank you for the response.

Yes, the resolution is at 0.468 x 0.468 x 1 mm. I might try with a stronger system to compensate.

I read in the --help notice under recon-surf that going below .7 would be experimental - how big of a difference/error in accuracy is expected from files with a higher resolution compared to .7-1mm ?

from fastsurfer.

.468 is far below .7 and I expect it to fail completely. Our code will interpolate to the smallest voxel dimension (so here it will be .468 isotropic) which is why it gets so large. I would recommend you run at .7 isotropic via --vox_size .7

from fastsurfer.

also is this 3T? The network has never seen a 7T image so that is also very experimental and would require at least some heavy bias field removal as preprocessing.

from fastsurfer.

Thank you for the response and advice, I will try running the image at .7 and see how it goes; and yes the image is in 3T.

Will close now

from fastsurfer.