Using Setfit for similarity classification about setfit HOT 5 OPEN

Hi @castafra, Thanks for your question. Currently Setfit does not support two sentences similarity classification tasks and it's not trivial to apply SetFit for that. It will probably require changes within the Sentence Transformer itself. We do have plans to extend SetFit and one of the options is to examine similarity classification task. Having said that, i did play a bit with it, as a workaround, I simply concatenated sentence1 and sentence2 and input pairs of 2 concatenated sentences to the contrastive training process. The results weren't so good. SetFit was on-par with the baseline model (a 'standard' BERT cross encoder) when using few-shot data (8,16,32 samples). The bottom line is that this workaround is not recommended.

from setfit.

I also work on the similarity classification task. I think it would be great to extend the framework for that task.
It seems setFit uses the underlying sentence_transformer library, right? FWIW, the example shows how to use sentence_transformer for similarity learning - https://www.sbert.net/examples/training/sts/README.html . I'm wondering whether we just need to implement a sklearn-learn classifier model to take two fine-tuned embeddings as inputs to predict the label.

Other references:

1. https://keras.io/examples/vision/metric_learning_tf_similarity/
2. https://keras.io/examples/vision/siamese_network/

from setfit.

Hi @castafra thanks for your interest in our work! I know @orenpereg ran some experiments on 2-sentence tasks like GLUE, but it wasn't entirely clear how one should create the text/label triples one needs for the contrastive learning step.

Perhaps he can share some more details on what worked / didn't work :)

from setfit.

Hi @orenpereg , could you share any feedbacks on your experiments ?
Thanks

from setfit.

Hi @castafra, Thanks for your question. Currently Setfit does not support two sentences similarity classification tasks and it's not trivial to apply SetFit for that. It will probably require changes within the Sentence Transformer itself. We do have plans to extend SetFit and one of the options is to examine similarity classification task. Having said that, i did play a bit with it, as a workaround, I simply concatenated sentence1 and sentence2 and input pairs of 2 concatenated sentences to the contrastive training process. The results weren't so good. SetFit was on-par with the baseline model (a 'standard' BERT cross encoder) when using few-shot data (8,16,32 samples). The bottom line is that this workaround is not recommended.

This is a bummer :( I was so excited that I could few shot improve postal address comparisons... :D

from setfit.