Large values of casual loss about dalm HOT 3 CLOSED

Seems like something is wrong! What was the batch size?

from dalm.

I found an error, now the loss is around 10 at the start of the training. I made some modifications to your code, so the mistake was mine. When computing the loss, I used the sum over tokens instead of the mean, similar to the original training code in the hf RAG repository. With the sum, it is expected to have a large loss which depends on the number of tokens. So, what is the reasoning behind that? Did I misunderstand it somehow?

By the way, as far as I am aware, your code does not allow for training only the query encoder and not the context encoder, as in the original RAG paper. Are there any plans to add this feature in the future?

from dalm.

@yashkens Yeah sum will give you large values and can mess up the training.

In this repo, we didn't use a duel encoder for a retriever.

from dalm.