code release about raat HOT 6 OPEN

Thanks for your interest and appreciation!

The code and collected data have now been uploaded. Feel free to use them, and let us know if you have any questions or feedback.

from raat.

感谢开源代码！

我在运行的时候产生了OutOfMemeryError的报错，但是我是按照readme中的bash跑的，此时per_device_train_batch_size也设为了1，所以有点奇怪。

我发现目前开源的代码中没有使用半精度微调，是否是这个原因？

from raat.

It might not be for this reason.

Although the batch_size in the script is set to 1, due to the special training method of RAAT, the actual batch_size during training is 4 because one query corresponds to 4 augmented samples. This might lead to OOM issues (you can check tuner/trainer/raat_processor.py). In the experimental setup described in our paper, we used 4 A100 GPUs with 80G memory each and deepspeed zero2. Here, we suggest that you modify the source code to reduce the types of retrieval noise. For example, one query corresponds to only 2 augmented samples: one Golden retrieval and one Golden retrieval + Random retrieval noise.

from raat.

Thanks for your prompt reply.

I also used 4 A100 GPUs, each with 80G memory.

I know that 4 augmented samples will cause the batch size to increase by 4 times, which may be the cause of OOM. However, I did not find where you used deepspeed zero2. Am I missing something?

from raat.

The source code does not provide a deepspeed YAML script. You can use 'accelerate config' command or write a YAML to implement Zero2.

from raat.

thanks. I will give it a try.

from raat.