tshi04 / leafnats Goto Github PK

Hello.
After training the pointer_generator_model with the CNN/DM data that was provided in #1 using the command:
python run.py --data_dir load_data/sum_data_rc_cnndm/

I tried to execute the validation task:
python run.py --task validate --data_dir load_data/sum_data_rc_cnndm/

In principle, it seems to work. However, the execution doesn't terminate. Here's the output of the program:
The vocabulary size: 50000
{}
{'decoder2proj': Linear(in_features=256, out_features=128, bias=False),
'embedding': natsEmbedding(
(embedding): Embedding(50000, 128)
(proj2vocab): Linear(in_features=128, out_features=50000, bias=True)
),
'encoder': natsEncoder(
(encoder): LSTM(128, 256, batch_first=True, bidirectional=True)
),
'encoder2decoder': natsEncoder2Decoder(
(encoder2decoder): Linear(in_features=512, out_features=256, bias=True)
(encoder2decoder_c): Linear(in_features=512, out_features=256, bias=True)
),
'pgdecoder': PointerGeneratorDecoder(
(rnn_): LSTMCell(384, 256)
(encoder_attn_layer): AttentionEncoder(
(attn_en_in): Linear(in_features=512, out_features=256, bias=True)
(attn_de_in): Linear(in_features=256, out_features=256, bias=False)
(attn_cv_in): Linear(in_features=1, out_features=256, bias=False)
(attn_warp_in): Linear(in_features=256, out_features=1, bias=False)
)
(decoder_attn_layer): AttentionDecoder(
(attn_en_in): Linear(in_features=256, out_features=256, bias=True)
(attn_de_in): Linear(in_features=256, out_features=256, bias=False)
(attn_warp_in): Linear(in_features=256, out_features=1, bias=False)
)
(attn_out): Linear(in_features=1024, out_features=256, bias=True)
(pt_out): Linear(in_features=1152, out_features=1, bias=True)
)}
Validate *_34_17951.model
The number of batches (test): 836
[>>>>>>>>>>>>>>>>>>>>>>>>>]100%
model=34_17951, loss=2.6192305088043213, time=6.445841550827026

Please note that I've inserted in ../nats_results only those .model files that have 34_17951. The contents of the directory, while the validation task is still running, are:
ll nats_results
total 59376
drwxrwxr-x 4 dvillanova dvillanova 4096 abr 20 18:03 ./
drwxr-xr-x 19 dvillanova dvillanova 4096 abr 20 18:14 ../
drwxrwxr-x 2 dvillanova dvillanova 20480 abr 20 18:03 batch_validate_16/
-rw-rw-r-- 1 dvillanova dvillanova 131494 abr 20 18:03 decoder2proj_34_17951.model
-rw-rw-r-- 1 dvillanova dvillanova 51400747 abr 20 18:03 embedding_34_17951.model
-rw-rw-r-- 1 dvillanova dvillanova 1051540 abr 20 18:03 encoder2decoder_34_17951.model
-rw-rw-r-- 1 dvillanova dvillanova 3163369 abr 20 18:03 encoder_34_17951.model
drwxrwxr-x 2 dvillanova dvillanova 4096 abr 20 18:03 model/
-rw-rw-r-- 1 dvillanova dvillanova 81 abr 20 18:03 model_validate.txt
-rw-rw-r-- 1 dvillanova dvillanova 5002832 abr 20 18:03 pgdecoder_34_17951.model

When ending manually the task (Ctrl + C) I get the following traceback:
^CTraceback (most recent call last):
File "run.py", line 1, in
from pointer_generator_network.main import *
File "/home/dvillanova/LeafNATS/pointer_generator_network/main.py", line 119, in
model.validate()
File "/home/dvillanova/LeafNATS/LeafNATS/engines/end2end.py", line 251, in validate
'..', 'nats_results', sorted(list(self.train_models))[0]+'*.model'))
File "/home/dvillanova/anaconda3/lib/python3.7/glob.py", line 20, in glob
return list(iglob(pathname, recursive=recursive))
File "/home/dvillanova/anaconda3/lib/python3.7/glob.py", line 72, in _iglob
for name in glob_in_dir(dirname, basename, dironly):
File "/home/dvillanova/anaconda3/lib/python3.7/glob.py", line 80, in _glob1
names = list(_iterdir(dirname, dironly))
File "/home/dvillanova/anaconda3/lib/python3.7/glob.py", line 122, in _iterdir
for entry in it:
KeyboardInterrupt

I run "from nats.headline2_summary2_app.main_app import * " and I used the pretrained model the repo owner provided, the same 1_in.txt file but got two different result as follows in attachments. Is there anything wrong in the repo?
1_in.txt
cpu_result.txt
gpu_result.txt

Hello,
I am working on Abstractive Text Summarization and I am facing multiple issues such as:

ISSUE #1 - RAM
I was trying to set it up on AWS t2.micro, but spaCy couldn't install due to low RAM (1 GB). Upgraded to 2 GB RAM and then spaCy installed successfully.
Later I was trying to run the Stanford-CoreNLP Server to test it, it did run but it couldn't execute any sentences and crashed with this error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000075072e000, 661209088, 0) failed; error='Not enough space' (errno=12)
Which I am guessing is a RAM error...
So in order to run this model, what amount of RAM is required?
What OS and hardware specifications (Graphic Card, etc.) would you recommend?

ISSUE #2 - Dataset
When I was setting up the "Bytecup2018" dataset, on the following command,
python3 tokenize.py --input bytecup.corpus.train.1.txt --output new1.txt
I got this error:
ImportError: cannot import name 'StanfordCoreNLP'
Which means that StanfordCoreNLP wasn't installed properly. Do we have to use the pywrapper version of StanfordCoreNLP? Because the link that you have given consists of java files. How should I go about it?

In order to run the model, do I have to install all three datasets- CNN/Dailymail, Newsroom and Bytecup?

ISSUE #3 - Pre-trained Model
Where do I place the pre-trained model?

I run the pointer-generator network of C10110 on Newsroom summary task with hyper parameters as following:
n_epch = 20
batch_size = 16
attn_method = luong_concat
repetition = vanilla

But the final result is ROUGE-1 = 36.75, ROUGE-2 = 26.06, ROUGE-L = 33.85

Could you please tell me about your experimental settings?

https://github.com/google-research/google-research/tree/master/rouge

I used this before, which is built with python fully. The usage seems straightforward (there's no need to set up in command lines, and etc.) The rouge that you have used seems to be with quite a bit burden.

RuntimeError: Error(s) in loading state_dict for natsEncoder:
size mismatch for encoder.weight_ih_l0: copying a param with shape torch.Size([1024, 128]) from checkpoint, the shape in current model is torch.Size([1024, 512]).
size mismatch for encoder.weight_ih_l0_reverse: copying a param with shape torch.Size([1024, 128]) from checkpoint, the shape in current model is torch.Size([1024, 512]).

Note:
Thanks for your work.
I reduce the amount of “bytecup” dataset, when i finish training the model,there are 35 files for every part.
I don`t know how to find the "embedding_base.model" and "encoder_base.modl", So i choose "embedding_0_0.model" and "encoder_0_0.model",then rename files.
Then i choose the "xxx_34.model" of every part as the end-use model, then rename them as "xxx_bytecup_title.model". But when i run "main_app.py" in the folder of headline2_summary2_app,there are the above Error. I debuug the code of generation model all the time,but can not find the where is the matrix [1024, 512].

I'm looking forward to your reply。Thank you again.

======================================================
Note:

Have solved the question.
I need to use "model_app.py" in the folder of "pointer_generator_network" rather than "headline2_summary2_app".

Cant run the required files because of missing nats dependecy

Traceback (most recent call last): File "run.py", line 1, in <module> from pointer_generator_network.main import * File "/home/yash/Desktop/LeafNATS-master/pointer_generator_network/main.py", line 119, in <module> model.test() File "/home/yash/Desktop/LeafNATS-master/LeafNATS/engines/end2end.py", line 362, in test self.build_vocabulary() File "/home/yash/Desktop/LeafNATS-master/pointer_generator_network/model.py", line 38, in build_vocabulary vocab2id, id2vocab = construct_vocab( File "/home/yash/Desktop/LeafNATS-master/LeafNATS/data/utils.py", line 30, in construct_vocab with open(file_, 'r') as fp: FileNotFoundError: [Errno 2] No such file or directory: '../sum_data/vocab'

I got an error during running run.py script, with following error messages:

FileNotFoundError: [Errno 2] No such file or directory: '../model/embedding_base.model'

Hello. I would like to train a model with the CORD-19 dataset by doing transfer learning from a model trained with the CNN/DM dataset (not from the proposed base model, I already have some results with that). I have been training and validating a model and I have my *_24_20900.model files in a folder. What should I do next? Should I put those files in a specific folder? Is there any argument to specify which .model files to take as base for transfer learning?

Thanks in advance!

Would you consider it to be better if a Python package could wrapped up? It's a bit lazy to ask for this kind of package, but guess that you can handle it.

Something like:

import leafnats #or
from leafnats import pgen #...
model = pgen(.., .., ..)

Hello Tian Shi,
Thanks for sharing the project!

Is it easy to visualize attention like this?

BTW, I pasted following content into the demo website. The summary/highlights is exact copy of the source text. Is this because the text is too short? And why can't I input one sentence for abstractive summary?

Evernote helps you focus on what matters most and have access to your information when you need it. Input typed notes or scan handwritten notes. Add to-do’s, photos, images, web pages, or audio ... and it’s all instantly searchable. Create notebooks, organizers, planners. Organize notes any way you want and share with anyone. And Evernote syncs your notes and notebooks across your devices so your information is always with you, everywhere you go.

"Use Evernote as the place you put everything … Don’t ask yourself which device it’s on—it’s in Evernote" – The New York Times

“When it comes to taking all manner of notes and getting work done, Evernote is an indispensable tool.” – PC Mag