Handling Negative Pairs in Fine-Tuning of CLIP Models about open_clip HOT 1 CLOSED

@doramasma be careful not to confuse triplet loss with contrastive. Triplet losses require an anchor in addition to postive and negative. Here we just have positive and negative pairings based on the image-text pairs in the dataset. And yes the batching does impact performance quite significantly for various reasons. As long as most of the time the positive pairs are true positives, and the negatives in the batch don't have too many false negatives (I feel this is a bigger confounder than worrying about the case you descripe across batches) there should be enough signal to learn from.

As for what's efficient, there have been loads of papers suggesting alternative forumulations trying to address issues/efficiency with the 'basic' formulation of the loss, making it more sample efficient, doing better with smaller global batch sizes, etc... but simple scales and many of those ideas weren't worth the overhead or just didn't perform at scale.

from open_clip.