yanzhangnlp / is-bert Goto Github PK

Hi,

Thanks very much for releasing your code!

Was wondering if it is worth training IS-BERT with a fine-tuned hugging face model rather than with an out-of-the-box hugging face model?

So, if I want to train IS-BERT on reddit comments, is it better to:

a.) Fine-tune 'bert-base-uncased' on reddit comments and then use embeddings from fine-tuned model to train IS-BERT

or

b.) Directly train IS-BERT on reddit comments using 'bert-base-uncased' embeddings without first fine-tuning 'bert-base-uncased' on reddit comments?

When I ran the code, I found that 'sentence_lengths'was not available in rep。why?Thank you!

Hi, since you treat each sentence and its local context representations as positive examples, and treat all the local context representations from other sentences as negative examples, like what we usually do in contrastive learning, why do you choose MI as loss function instead of conventional CL loss like InfoNCE? Is MI better than InfoNCE in this scenario? Thanks!

Hi,

First of all thank you for sharing your code!

I am wondering if there is a way to get an embedding for a new sentence after i trained the model with my data. model.encode(sentence) seems to be doing this. But I tested it using a sentence from the sentences I trained the model with and the embeddings differ. Am I doing something wrong?

Thank you in advance for your help!

Thank you for the brilliant work and sharing it on the web.
Seems all the code are well organized conveniently for users.

Reading thru the docs about training and the paper did not resolve the question of mine.

Is the initialization of the BERT module in IS-BERT is bert-base-uncased or
did you wanted to just specify the architectural aspect but not the pre-trained weight?

Is the discriminator Tw in this paper dot product operation?

hello,what these measures such as 'GAN','JSD' and so on mean?