CUDA out of memory when using the network to train about mtfaa-net HOT 3 CLOSED

After some debugging, it seems this line:

t_score = torch.einsum('bcft, bcfy->bfty', [qt, kt]) / (self.d_c**0.5)

in asa.py file occupied a lot of memory at the first TFCM+ASA modules after band merging. I would see memory usage jumped from about 2GB to 10GB, and that cannot be deleted or cache emptied for some reason.

from mtfaa-net.

Thanks for your attention,

I think 30-second may be too long, especially using attention structure which needs more memory for long audio. You could remove the t-attention or use shorter segments.

from mtfaa-net.

Ahhh, thank you for the suggestions! I will try them out

from mtfaa-net.