rohithreddy024 / text-summarizer-pytorch Goto Github PK

Pytorch implementation of "A Deep Reinforced Model for Abstractive Summarization" paper and pointer generator network

Python 100.00%

pytorch text-summarization mle-training rl-training beam-search

text-summarizer-pytorch's Introduction

Text-Summarizer-Pytorch

Combining A Deep Reinforced Model for Abstractive Summarization and Get To The Point: Summarization with Pointer-Generator Networks

Model Description

LSTM based Sequence-to-Sequence model for Abstractive Summarization
Pointer mechanism for handling Out of Vocabulary (OOV) words See et al. (2017)
Intra-temporal and Intra-decoder attention for handling repeated words Paulus et al. (2018)
Self-critic policy gradient training along with MLE training Paulus et al. (2018)

Prerequisites

Pytorch
Tensorflow
Python 2 & 3
rouge

Data

Download train and valid pairs (article, title) of OpenNMT provided Gigaword dataset from here
Copy files train.article.txt, train.title.txt, valid.article.filter.txtand valid.title.filter.txt to data/unfinished folder
Files are already preprcessed

Creating `.bin` files and vocab file

The model accepts data in the form of .bin files.
To convert .txt file into .bin file and chunk them further, run (requires Python 2 & Tensorflow):

python make_data_files.py

You will find the data in data/chunked folder and vocab file in data folder

Training

As suggested in Paulus et al. (2018), first pretrain the seq-to-seq model using MLE (with Python 3):

python train.py --train_mle=yes --train_rl=no --mle_weight=1.0

Next, find the best saved model on validation data by running (with Python 3):

python eval.py --task=validate --start_from=0005000.tar

After finding the best model (lets say 0100000.tar) with high rouge-l f score, load it and run (with Python 3):

python train.py --train_mle=yes --train_rl=yes --mle_weight=0.25 --load_model=0100000.tar --new_lr=0.0001

for MLE + RL training (or)

python train.py --train_mle=no --train_rl=yes --mle_weight=0.0 --load_model=0100000.tar --new_lr=0.0001

for RL training

Validation

To perform validation of RL training, run (with Python 3):

python eval.py --task=validate --start_from=0100000.tar

Testing

After finding the best model of RL training (lets say 0200000.tar), evaluate it on test data & get all rouge metrics by running (with Python 3):

python eval.py --task=test --load_model=0200000.tar

Results

Rouge scores obtained by using best MLE trained model on test set:
scores: {
'rouge-1': {'f': 0.4412018559893622, 'p': 0.4814799494024485, 'r': 0.4232331027817015},
'rouge-2': {'f': 0.23238981595683728, 'p': 0.2531296070596062, 'r': 0.22407861554997008},
'rouge-l': {'f': 0.40477682528278364, 'p': 0.4584684491434479, 'r': 0.40351107200202596}
}
Rouge scores obtained by using best MLE + RL trained model on test set:
scores: {
'rouge-1': {'f': 0.4499047033247696, 'p': 0.4853756369556345, 'r': 0.43544461386607497},
'rouge-2': {'f': 0.24037014314625643, 'p': 0.25903387205387235, 'r': 0.23362662645146298},
'rouge-l': {'f': 0.41320241732946406, 'p': 0.4616655167980162, 'r': 0.4144419466382236}
}
Training log file is included in the repository

Examples

article: russia 's lower house of parliament was scheduled friday to debate an appeal to the prime minister that challenged the right of u.s.-funded radio liberty to operate in russia following its introduction of broadcasts targeting chechnya .
ref: russia 's lower house of parliament mulls challenge to radio liberty
dec: russian parliament to debate on banning radio liberty

article: continued dialogue with the democratic people 's republic of korea is important although australia 's plan to open its embassy in pyongyang has been shelved because of the crisis over the dprk 's nuclear weapons program , australian foreign minister alexander downer said on friday .
ref: dialogue with dprk important says australian foreign minister
dec: australian fm says dialogue with dprk important

article: water levels in the zambezi river are rising due to heavy rains in its catchment area , prompting zimbabwe 's civil protection unit -lrb- cpu -rrb- to issue a flood alert for people living in the zambezi valley , the herald reported on friday .
ref: floods loom in zambezi valley
dec: water levels rising in zambezi river

article: tens of thousands of people have fled samarra , about ## miles north of baghdad , in recent weeks , expecting a showdown between u.s. troops and heavily armed groups within the city , according to u.s. and iraqi sources .
ref: thousands flee samarra fearing battle
dec: tens of thousands flee samarra expecting showdown with u.s. troops

article: the #### tung blossom festival will kick off saturday with a fun-filled ceremony at the west lake resort in the northern taiwan county of miaoli , a hakka stronghold , the council of hakka affairs -lrb- cha -rrb- announced tuesday .
ref: #### tung blossom festival to kick off saturday
dec: #### tung blossom festival to kick off in miaoli

References

pytorch implementation of "Get To The Point: Summarization with Pointer-Generator Networks"

text-summarizer-pytorch's People

Contributors

Stargazers

Watchers

Forkers

mastercaojie mqrshiyan praveengupta18 ruotianluo wattba vyraun singhranjodh derenlei legendtianjin alex-fabbri williamkulp sotheara-leang santhoshsthanikam jbot19k-fork may-sunshine tzoof huangxiaolist caoxu915683474 hellomaxwell zhangxuann jessie0624 jascorea 1204271075 kanikasaini xw-jia harx11 bigapartmentsin nikolausn sayan3710 erick093 architect1st daywatch dixonch yashwanthv19 georgetao 127 rocassius mohitjuneja projectifyofficial gongcq tintinxue1 humdingers jiminsun pptt168 a396198172 gaominwu psqqq19 meghraj-webllisto da-southampton heartingu junaraki bobycv06fpm cm107 anything4share mcao516 neuztb robotse7en xueqing-chen tangashley lsq960124 thdai2000 shuishen112 v-ipul cliuxinxin meizhiju daniel4se garipovagz pedrocarvalho98 berylv587 jus1mple theanhle techthiyanes aqhali pushkar-bhardwaj-iitk

text-summarizer-pytorch's Issues

python2 not support tensorflow

first step python make_data_files.py
but make_data_files.py need import tensorflow, code like:
from tensorflow.core.example import example_pb2
how to solve this problem?

Results on CNN/DM dataset?

Has anyone run this model on the CNN/DM dataset before? and what's the result?

Thanks!

determine max and min summary lengths

I want to generate longer summaries from the data and to do this I changed the max and min_decode_length parameters in the config file. However, it doesn't seem to be changing all that much. Some help here would be much appreciated!

No such file file or directory "data/vocab"

train.py gives this error while trying to read data/vocab

There is no vocab file in data directory,

can someone help?

RL mode requires much more GPU memory

I find one empty GPU, there is no more error, however, it still confused me that why RL mode requires much more memory?

Hi, Thanks for your excellent job, But when I start RL training, some errors occurs as follows"

Traceback (most recent call last):
  File "train.py", line 316, in <module>
    train_processor.train_iters()
  File "train.py", line 274, in train_iters
    mle_loss, r = self.train_one_batch(batch, iter)
  File "train.py", line 238, in train_one_batch
    enc_batch_extend_vocab, batch.art_oovs, greedy=True)
  File "train.py", line 164, in train_batch_rl
    sum_temporal_srcs, prev_s)
  File "/users4/xiachongfeng/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/xiachongfeng/Summ_entail/model.py", line 142, in forward
    ct_e, attn_dist, sum_temporal_srcs = self.enc_attention(st_hat, enc_out, enc_padding_mask, sum_temporal_srcs)
  File "/users4/xiachongfeng/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/xiachongfeng/Summ_entail/model.py", line 52, in forward
    et = self.W_h(h)  # bs,n_seq,2*n_hid
  File "/users4/xiachongfeng/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/xiachongfeng/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/users4/xiachongfeng/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1026, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: out of memory

Could you please help me?

RuntimeError: CUDA error: device-side assert triggered while running MLE + RL training

I am getting following error while running MLE + RL training

Traceback (most recent call last): File "train.py", line 270, in <module> train_processor.trainIters() File "train.py", line 238, in trainIters mle_loss, r = self.train_one_batch(batch, iter) File "train.py", line 202, in train_one_batch mle_loss = self.train_batch_MLE(enc_out, enc_hidden, enc_padding_mask, context, extra_zeros, enc_batch_extend_vocab, batch) File "train.py", line 80, in train_batch_MLE use_gound_truth = get_cuda((T.rand(len(enc_out)) > 0.25)).long() #Probabilities indicating whether to use ground truth labels instead of previous decoded tokens File "/home/exa00117/Project/textSummerization/rnn/Text-Summarizer-Pytorch-master/train_util.py", line 7, in get_cuda tensor = tensor.cuda()

Could you please help resolve it?

No tar file produced after first step

After executing "python train.py --train_mle=yes --train_rl=no --mle_weight=1.0", I tried "python eval.py --task=validate --start_from=0005000.tar" but got "ValueError: '0005000.tar' is not in list".

I left the first step to execute for many hours (on a 12vCPUs and 1 Titan GPU machine), it didn't end so I manually triggered a keyboard interrupt after 8 hours.

I can't seem to find any tar files in any of my subdirectories, was the first step supposed to generate these?

MLE loss becomes zero immediately after first step

I was trying to run the regular training process with MLE loss on CNN/DM dataset, before attempting any modifications. However, while the code in itself is running fine, the loss becomes zero immediately after the first step, whatever the batch size.

I was wondering if you would be able to tell why such a situation might occur.
Thanks in advance.

cuda runtime error (59) : device-side assert triggered during training

Hello!

I've been trying to use your code for training on Gigaword dataset, but after some iterations it starts to throw quite non-descriptive error: cuda runtime error (59) : device-side assert triggered.

Following suggestion from https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-59/750/3 and setting CUDA_LAUNCH_BLOCKING=1, the backtrace of the error shows it happening in this line: https://github.com/rohithreddy024/Text-Summarizer-Pytorch/blob/master/train.py#L89 .

The training is done in Colab with GPU support. I am using default config (as in the code on GitHub). The moment when the error happens seems to be rather undetermined - first time happened around 45000 iteration, but went past that after few retries, but now it happens always after 55000 iteration.

Have you seen an error like this? Do you know what can cause it?

I've tried to use PDB but after first CUDA error happens, calling any of the decoder-produced variables causes it as well. Do you know of any other ways that would help with debugging it?

Thank you!

how to pre-process CNN/DM？

could you please tell me how to use your framework with CNN/DM dataset？
I don't know how to create .bin files with CNN/DM?
Please inspire me with something, tutorial or else.

Training time

So I'm trying to train this model on my GPU, and it takes about 1800 seconds for 1000 steps so about 10 days training time. Could you tell me if you had a similar experience if not, how long did it take you to train the model?

RL results are worse than Mle results in Rouge-1,2

hi, we found the best model (0050000.tar) for rl training after mle training, Although the rouge-L score improved, but the rouge-1 and rouge-2 score became very bad .
we show the eval:(we use rouge-1 for evaluation).
mle (official testset):
Training mle: yes, Training rl: no, mle weight: 1.00, rl weight: 0.00
intra_encoder: True intra_decoder: True
0005000.tar rouge_1: 0.3174
0010000.tar rouge_1: 0.3249
0015000.tar rouge_1: 0.3289
0020000.tar rouge_1: 0.3325
0025000.tar rouge_1: 0.3331
0030000.tar rouge_1: 0.3357
0035000.tar rouge_1: 0.3379
0040000.tar rouge_1: 0.3355
0045000.tar rouge_1: 0.3382
0050000.tar rouge_1: 0.3426
0055000.tar rouge_1: 0.3384
0060000.tar rouge_1: 0.3339
0065000.tar rouge_1: 0.3410
0070000.tar rouge_1: 0.3408
0075000.tar rouge_1: 0.3425
0080000.tar rouge_1: 0.3384
0085000.tar rouge_1: 0.3362
0090000.tar rouge_1: 0.3424
0095000.tar rouge_1: 0.3377
0100000.tar rouge_1: 0.3361
0105000.tar rouge_1: 0.3357
0110000.tar rouge_1: 0.3389
0115000.tar rouge_1: 0.3374
0120000.tar rouge_1: 0.3341
0125000.tar rouge_1: 0.3357
0130000.tar rouge_1: 0.3377
0135000.tar rouge_1: 0.3317
0140000.tar rouge_1: 0.3321
0145000.tar rouge_1: 0.3349
0150000.tar rouge_1: 0.3363
rl (official testset):
in_rl=yes --mle_weight=0.0 --load_model=0050000.tar --new_lr=0.0001
Training mle: no, Training rl: yes, mle weight: 0.00, rl weight: 1.00
intra_encoder: True intra_decoder: True
Loaded model at data/saved_models/0050000.tar
0050000.tar rouge_1: 0.3426
0055000.tar rouge_1: 0.2522
0060000.tar rouge_1: 0.2520
0065000.tar rouge_1: 0.2549
0070000.tar rouge_1: 0.2550
0075000.tar rouge_1: 0.2547
0080000.tar rouge_1: 0.2584
0085000.tar rouge_1: 0.2576
0090000.tar rouge_1: 0.2543
0095000.tar rouge_1: 0.2567
0100000.tar rouge_1: 0.2562
0105000.tar rouge_1: 0.2556
0110000.tar rouge_1: 0.2547
0115000.tar rouge_1: 0.2575
0120000.tar rouge_1: 0.2543
0125000.tar rouge_1: 0.2581
0130000.tar rouge_1: 0.2534
0135000.tar rouge_1: 0.2533
0140000.tar rouge_1: 0.2526
0145000.tar rouge_1: 0.2511
0150000.tar rouge_1: 0.2547

mle result:
0075000.tar scores: {'rouge-1': {'f': 0.3424728366572667, 'p': 0.39166721241721236, 'r': 0.31968494072078807}, 'rouge-2': {'f': 0.1732520206640223, 'p': 0.19845553983053968, 'r': 0.1623725413112666}, 'rouge-l': {'f': 0.32962985739519235, 'p': 0.3758193750693756, 'r': 0.3075168451832533}}

rl result:
0080000.tar scores: {'rouge-1': {'f': 0.2574669041724543, 'p': 0.21302155489848726, 'r': 0.34803503077209935}, 'rouge-2': {'f': 0.11896310475645827, 'p': 0.09758671687502977, 'r': 0.16587082443700088}, 'rouge-l': {'f': 0.35379459020991105, 'p': 0.39799812070645335, 'r': 0.33855028225319733}}

0125000.tar scores: {'rouge-1': {'f': 0.25674349158898563, 'p': 0.21440196978373974, 'r': 0.34277860517537473}, 'rouge-2': {'f': 0.11907341598225046, 'p': 0.09900864566338015, 'r': 0.16306397570581008}, 'rouge-l': {'f': 0.35462601354567735, 'p': 0.40579230645897313, 'r': 0.33368591052575747}}
thanks for your help!

Accessing examples

What I need to do to get access to output of decoder?

intra decoder code doesn't match paper

The intra decoder code in model.py (lines 124 to 133) does not match either paper. The referenced pointer-generator paper, eqn 1, uses encoder hidden states, current decoder state, and a bias term. There is no mention of list of previous decoder states. In the other paper, "deep reinforced model...", there is use of prev decoder states, but not in the way you have it in the code. Can you point to the specific formula that's behind your code?

Error in beam_search.py

self.hid_h = h[beams_order], while beam_orders are not index

No coverage for pointer generator?

There seems to be no coverage loss implemented for the pointer generator, is there any way i could go about implementing it?

the problem of x_t

I can not understand the code in the 76 line of train.py
why use start_id to fill the x_t ?
x_t = get_cuda(T.LongTensor(len(enc_out)).fill_(self.start_id))

How long it takes in a non GPU system

Hi Rohit,

Thanks for the repo, I have a non-GPU machine with 32 GB RAM, Windows 10 OS, I ran the train.py and it is running for more than 12 hours now and just the first iteration is running.
i.e iter: 1000 mle_loss: 4.652 reward: 0.0000

Just wondering if the code is running correctly? and also wanted to ask, how long it will take for a complete execution?
Is GPU mandatory to run the training ?

article portion is not there

Hi,
I have done the training on your git repo but code where you gave article and ref and dec is printing is not there i guess ?

Please help for the same.

please provide the best trained model

Hi, could you provide the best trained model?

Type error in line 151

Hi Guys,
I am getting the following error
pointer_summarizer-master/data_util/data.py", line 149, in abstract2sents
while True:
TypeError: argument should be integer or bytes-like object, not 'str'

pointer_summarizer-master/data_util/data.py", line 151, in abstract2sents
start_p = abstract.index(SENTENCE_START, cur)
Any idea on how ti fix the error?

Beam Search get_current_state deletes OOV tokens

In Beam.get_current_state() in beam_search.py, setting OOV tokens to UNK also changes self.tokens, as self.tokens[:, -1] returns a view to the same underlying memory.

This leads to a bug, where beam_search never decodes OOV words.

Fix: Use .clone(), to create a copy of the tensor.

tokens = self.tokens[:, -1].clone()

the output for beams_order is tensor([0.0557, 0.0172, 0.0002, 0.0270], device='cuda:0')