Contrastive losses and marginalization connection about dalm HOT 2 CLOSED

1. Well there are many ways to do the contrastive loss. What we follow is more or less this. Pretty much what the CLIP authors used and almost all these recent embedding transformer authors such as E5.

2. Well we need to somehow combine the doc-scores with the next word prediction right ? why would you think adding the log prob won't do any difference ?
   I took the equation from here.

from dalm.

Hey @shamanez

Thank you for the replies.

re:1 of course, there are many ways to contrastive losses. I'll read through the linked paper

re:2 I think the theoretical aspect of adding the logprobs make sense. My question was based around the runtime values I see, seem to be adding the same increment to all values of answer_log_prob, to put my question explicitly

# shape: [102, 50257]
answer_log_prob = logprobs_logits[query_token_length - 1 :, :]

#  addition of shapes: [102,50257] + [1,1]   
marginalized_prob_sum = answer_log_prob + doc_logprobs

I think my seemingly faulty reasoning reasoned that adding the same value to all values of the distribution seems off. I'll have to work my way out of this, as I still don't understand what this means , in my head I'd expect the following

#  addition of shapes: [102,50257] + [1,50257]   
marginalized_prob_sum = answer_log_prob + doc_logprobs

It is very likely it's just my inexperience speaking. I'll work on this

I guess with those two question being answered. I think that's the end of my questions with the loss part and rest of the code flow for now

from dalm.