Cannot train the longformer-base-4096 in CNN/DM dataset about transformersum HOT 7 CLOSED

@ShibataGenjiro Thank you for the detailed bug report. The longformer does not support token_type_ids so you need to set the --no_use_token_type_ids option. I've pushed a change that will automatically enable this option for the longformer. I also opened huggingface/transformers#9111 to make the huggingface/transformers documentation more clear that the longformer does not support token_type_ids. Let me know how your training run goes. If you end up training the longformer on CNN/DM, I'd appreciate it if you open a pull request with a link to the model weights file so it can be added to the library.

from transformersum.

@HHousen Thank you very much. The longformer can be trained now. (But training is very slow, because I set the batch_size to 1 on single 3090 GPU. If I set a larger batch_size, OOM problem will occur.)

Anyway, you said that the longformer does not use the token_type_ids (segment_id in BERTSUM, I think).

Does this mean that longform only uses token embeddings and position embeddings as input? (while BERTSUM uses token embeddings, segment embeddings and position embeddings)

from transformersum.

@ShibataGenjiro Correct, the longformer only uses token embeddings and position embeddings while BERT uses token embeddings, segment embeddings, and position embeddings. This is because the longformer is based on RoBERTa, which is an improved version of BERT. Regarding the OOM issue, you can try setting --gradient_checkpointing to enable less memory consumption at the expense of a slower backward pass.

from transformersum.

@HHousen OK, I will try. Thank you for your patience in explaining!^^

from transformersum.

@HHousen OK, I will try. Thank you for your patience in explaining!^^

No problem 😄.

from transformersum.

Hi @HHousen I am having the exact set of errors when doing abstractive summarization. Is abstractive summarization with CNN/DM dataset not supported with longformer? I checked the changes that you made in #0729e1f08135a81f2a12062a248eb9ab557a0f6f but that does not seem to translate to abstractive summarization. Also, the option --no_use_token_type_ids does not seem to be a valid option for abstractive.

from transformersum.

Hi @HHousen I am having the exact set of errors when doing abstractive summarization. Is abstractive summarization with CNN/DM dataset not supported with longformer? I checked the changes that you made in #0729e1f08135a81f2a12062a248eb9ab557a0f6f but that does not seem to translate to abstractive summarization. Also, the option --no_use_token_type_ids does not seem to be a valid option for abstractive.

@thechargedneutron A seq2seq (text-to-text) model is needed for abstractive summarization (like t5, BART, etc). The longformer is just an encoder. It does not have a decoder. However, the LED exists for this exact purpose. Here is the huggingface/transformers documentation.

from transformersum.