❓ Question : Training - Evaluating discrepancy in Abstractive Summarization about unilm HOT 4 CLOSED

Hi @colanim ,

Only the encoder is pre-trained in a bidirectional manner, while the decoder is left-to-right, which is controlled by the attention mask matrix. So the fine-tuning process is the same as inference in terms of decoding.

-li

from unilm.

Thanks for the quick response @donglixp !

So if I understood well, for abstractive summarization there is 3 tasks :

1. left-to-right LM on the summary part for decoder
2. Bidirectional LM on the article part for the encoder
3. Extractive task based on the first token

Is it right ?

from unilm.

Because the source side has been given. During fine-tuning, we only compute generation loss for the decoder, which is similar to previous seq2seq models. In the paper, we added an extractive loss in the encoder side, but we didn't use it in the repo's example. The released checkpoint can achieve better results even without the extractive loss.

from unilm.

Ok so in the actual code there is only one loss, which is the generation loss for the decoder (so, left-to-right LM based on the summary).

Thank you very much for your answers !

from unilm.