I used the package in requirements.txt you provided in Python 3.7 environment.
But, I am encountering an issue where the GPU memory usage gradually increases during model training until it eventually leads to a CUDA out of memory error.
Thanks for your great job! I wonder if you can provide the example code on how to deploy the IntTower in real scenario? such as how to execute the multi-head faiss and maxsim in a parallel way.
Thank you for your great job. Could you provide the training script for the dataset of Amazon and Alibaba? We want to know more detail about your great job for following. Thank you very much!
First of all thanks a lot for your great article and for opening the code base.
I have a question regarding the model serving:
I understand that you create Faiss indices based on the multi-head latent representation of the items but how do you query them? Do you use the multi-head latent representation of the last layer of the user tower? And after retrieving the top K items, do you compute the Fe score to rerank the candidates?
As I known, faiss does not support 'max' operation.
Fot i-th layer user representaion, we will compute each head pairwise to get the similarity score, So we need to retrieve H^2 times?If there are L layers, eventually we need to retrieve L* H^2 times?
RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 23.99 GiB total capacity; 37.27 GiB already allocated; 0 bytes free; 37.33 GiB reserved in total by PyTorch)
I run this code on 24G GPU, this error always happened after epoch 2 whatever batch_size I set, is there anything wrong with my environment?