We have updated instructions on how to apply contrastive search on encoder-decoder models (e.g. BART and T5). about simctg HOT 8 CLOSED

@yxuansu Perfect thanks for the effort. This will help me a lot. I will contact you if I have any issues :)

from simctg.

@yxuansu Perfect thanks for the effort. This will help me a lot. I will contact you if I have any issues :)

You are welcome. Please free feel to contact if you meet any problem :-)

from simctg.

Have you used simCTG on tasks like summarization ? After reading the paper I think simCTG can be adopted to tasks like text-summarization. Since simCTG was designed for open-ended generation how can I adopt it for text summarization ?

Any tips or advices ?

from simctg.

Have you used simCTG on tasks like summarization ? After reading the paper I think simCTG can be adopted to tasks like text-summarization. Since simCTG was designed for open-ended generation how can I adopt it for text summarization ?

Any tips or advices ?

Hi,

I definitely think SimCTG can be applied to tasks like summarization or translation. My advice is to follow the same procedure as described in the paper: (1) first using contrastive training to train you summarization model (e.g. BART or T5); (2) use contrastive search to generate the result.

For the contrastive training on encoder-decoder models, I recommend you to read this blog (https://zenn.dev/kwashizzz/articles/ml-simctg-contrastive-framework). They provide a good tutorial and code implementation on applying contrastive training on the T5 model. After contrastive training, you can refer to our tutorial (https://github.com/yxuansu/SimCTG/tree/main/SimCTGEncDec) on how to apply contrastive search to your trained model.

Please let me know if you have any further questions.

from simctg.

in your instructions do you apply diverse contrastive search ? if not any hints on how to implement it ?

Also what is the effect of beam_width ?

from simctg.

in your instructions do you apply diverse contrastive search ? if not any hints on how to implement it ?

Also what is the effect of beam_width ?

No, we did not implement the diverse contrastive search. I think you can easily adapt the code by yourself by referring to the details here (

Regarding the beam width, it does not affect that much as compared with \alpha, but I recommend you to try different values of beam width to see the performance difference.

from simctg.

I was training BART for summarization using simCTG on custom dataset, the model does not seem to improve after first epoch. I do not use external metrics such as rogue, I am using validation loss to save the best model. Any recommendations to stabilise the training and improve the performance ?

from simctg.

I was training BART for summarization using simCTG on custom dataset, the model does not seem to improve after first epoch. I do not use external metrics such as rogue, I am using validation loss to save the best model. Any recommendations to stabilise the training and improve the performance ?

Can I see the learning log of the MLE loss and contrastive loss of your training process?

from simctg.