pliang279 / factorcl Goto Github PK

Local run(best score)
accuracy | MOSEI | MOSI | sarcasm | humor
FactorCL-SUP | 71.05 | 51.75 | 55.07 | 49.53
SupCon | 74.61 | 51.17 | 52.17 | 50.85

Reported
accuracy | MOSEI | MOSI | sarcasm | humor
FactorCL-SUP | 77.34 | 70.69 | 59.42 | 54.44
SupCon | 72.71 | 47.23 | 46.38 | 52.36

Hi authors,

I ran the notebook file(multibench_example.ipynb) without any modification and found significant gap compared to what's reported on the paper.

The numbers I wrote in the local run result are actually the best scores I could get from 5 to 10 independent runs, and the deviations were quite large ranging from 3 to 5% (compared to std of 0.2~ 2.8% from the paper)

Any idea what could went wrong with it?
Other experiments including Synthetic, IRFL were found to be reproducible.

Thanks in advance.

Thank you for your great work!
I have a question about a conditional loss in FactorCL-SSL case.
In IRFL_model.py Line 311, conditional CLUB loss is computed as follows:

self.club_x1x2_cond(torch.cat([self.linears_club_x1x2_cond[0](x1_embed),
                              self.linears_club_x1x2_cond[0](x1_embed)], dim=1), 
                    torch.cat([self.linears_club_x1x2_cond[1](x2_embed), 
                              self.linears_club_x1x2_cond[1](x2_embed)], dim=1))

However, I think that "embeds" should be concatenated with "aug_embeds" following Eq(8) in the paper, like:

self.club_x1x2_cond(torch.cat([self.linears_club_x1x2_cond[0](x1_embed), 
                              self.linears_club_x1x2_cond[0](x1_aug_embed)], dim=1), 
                    torch.cat([self.linears_club_x1x2_cond[1](x2_embed), 
                              self.linears_club_x1x2_cond[1](x2_aug_embed)], dim=1))

Since I'm a beginner in this field, I might have misunderstood something.
Is there a chance I might have misunderstood something?
Your response would be really helpful for me! Thank you.