Requesting some clarification regarding the core idea about cfvqa HOT 2 CLOSED

Thank you for your interest in our work!

1- In equations 11, 12, and 13 you are replacing the learned embeddings by a learnable constant, how this constant may be interpreted? what does it imply?
2- Why fixing this constant across the whole dimension, by multiplying it to ones?

The behind assumption is that if the image and question are blocked, the model would do a wild guess, i.e., each answer has the same probability to be selected. Therefore, we use the same constant to denote the same probability for all the candidate answers.

3- Following this,

       z_qkv = self.fusion(logits, q_pred, v_pred, q_fact=True,  k_fact=True, v_fact=True) # te
       z_q = self.fusion(logits, q_pred, v_pred, q_fact=True,  k_fact=False, v_fact=False) # nie
       logits_cfvqa = z_qkv - z_q

if we neglect the non-linearity (z = torch.log(torch.sigmoid(z) + eps)), (z_qkv - z_q) will be interpreted as (z_k + z_q + z_v) - (2C + z_q) which means we can just rely from the beginning on z_k + z_v and remove the QA branch?! I think I misunderstand something here :D

That's why we use the non-linearity to fuse the three outputs. Such non-linear design is important to avoid the trial solution.

4- Is it possible to replace the constant with other real example, such as augmented version of the input or something like that, what do u think?

We tried that in our implementation, e.g, using a mean feature as the input, but it does not work well. This idea may work for other tasks.

from cfvqa.

Thank you very much for your prompt response and for the clarifications!

from cfvqa.