yeliu918 / kg-bart Goto Github PK

View Code? Open in Web Editor NEW

157.0 3.0 18.0 212 KB

KG-BART: Knowledge Graph-Augmented BART for GenerativeCommonsense Reasoning

Python 100.00%

kg-bart's Introduction

KG-BART

KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning

0. Introduction

This is the official code base for the models in our paper on generative commonsense reasoning:

Ye Liu, Yao Wan, Lifang He, Hao Peng, Philip S. Yu. KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning. In AAAI 2021. (https://arxiv.org/abs/2009.12677) This code base is designed to reproduce our experiments in our paper, and can be used with other datasets. For any technical details, please refer to our paper. When using our code or the methods, please cite our paper: reference.

1. Knowledge Graph Grounding

Option 1: Download our processed data:

Option 2: Following our guidness and process from the original data: Download the following data to dataset direction:

Download the Conceptnet
Download the CommonGen Dataset, which require to fill the form

1.1 Preparing data

python reorder_src.py --dataset_dir "dataset"  --save_dataset_dir "dataset/save_dataset" 
--org_conceptnet "conceptnet-assertions-5.7.0.csv"  --save_conceptnet "conceptnet.csv"
python conceptnet.py --dataset_dir "dataset"  --save_dataset_dir "dataset/save_dataset"   
--OpenKE_dir "../OpenKE/benchmarks/CommonGen"  --save_conceptnet "conceptnet.csv"

1.2 Training Entity and Relation Embedding using TransE

cd OpenKE
git clone -b OpenKE-PyTorch https://github.com/thunlp/OpenKE
cd benchmarks/CommonGen
python n-n.py
python train_transe_CommonGen.py
cd ../KG_grounding
python entity_onehot.py --dataset_dir "dataset"  --save_dataset_dir "dataset/save_dataset"  
--OpenKE_dir "../OpenKE/benchmarks/CommonGen"  --save_conceptnet "conceptnet.csv"

2. Graph-Based Encoder-Decoder Modeling

2.0 Pre-training

Option 2.0.1 The following command shows how to pretrain our KG-BART model with the Conceptnet dataset created by ./get_data.sh as described above:

python pretrain_kgbart.py --data_dir ../dataset/commongen_data/commongen --output_dir ../output/Pretraining_KG 
    --log_dir ../log/Pretraining_KG2 --pretraining_KG --train_pretraining_num 400000 --val_pretraining_num 80000 
    --fp16 True --max_seq_length 32 --max_position_embeddings 64 --max_len_a 32 --max_len_b 64 --max_pred 64 
    --train_batch_size 60 --eval_batch_size 48 --gradient_accumulation_steps 6 --learning_rate 0.00001 
    --warmup_proportion 0.1 --label_smoothing 0.1 --num_train_epochs 10

Option 2.0.2 Download the Pre-training Weight

2.1 Train and evaluate CommonGen

The following command shows how to fine-tune our KG-BART model with the CommonGen dataset created by ./get_data.sh as described above:

python run_seq2seq.py --data_dir  ../dataset/commongen_data/commongen --output_dir ../output/KGBart
    --log_dir ../log/KGBart --model_recover_path ../output/Pretraining_KG/best_model/model.best.bin --fp16 True
    --max_seq_length 32 --max_position_embeddings 64 --max_len_a 32 --max_len_b 64 --max_pred 64
    --train_batch_size 60 --eval_batch_size 48 --gradient_accumulation_steps 6 --learning_rate 0.00001
    --warmup_proportion 0.1 --label_smoothing 0.1 --num_train_epochs 10

2.1 Test CommonGen

Finally, we can evalaute the models on the test set, by running the following command:

python decode_seq2seq.py --data_dir ../dataset/commongen_data/commongen --model_recover_path ../output/KGBart/best_model/model.best.bin
 --input_file ../dataset/commongen_data/commongen/commongen.dev.src_alpha.txt --output_dir ../output/KGBart/best_model/Gen
 --output_file model.best --split dev --beam_size 5 --forbid_duplicate_ngrams True

3. Evaluation

Download the Evaluation Package and follow the README: Evaluation

Dependencies

Python 3 (tested on python 3.6)
PyTorch (with GPU and CUDA enabled installation)
amp
tqdm

Acknowledgments

Our code is based on Huggingface Transformer. We thank the authors for their wonderful open-source efforts.

kg-bart's People

Contributors

Stargazers

Watchers

Forkers

yyht shubhampachori12110095 zwytop jucho2725 xc15071347094 lewis4188 dadaqingjian amy-hyunji jar2333 njcx-ai pengwei-iie morecry techthiyanes tony2016liang moziofmoon jihyelimm abdoiiii zedd-08

kg-bart's Issues

missing files

Please point out where the files of "conceptnet-assertions-5.7.0.csv" and "../OpenKE/benchmarks/CommonGen" are? When I ran reorder_src, the program asks these files.
Thanks.

ImportError: cannot import name 'AddedToken' from 'tokenizers'

tokenizers not found

I try to run run_seq2seq.py and I got below error. I can't find any file named tokenizers.py or any function named RobertaProcessing. Does this repository miss some files or I make something wrong?

Traceback (most recent call last):
File "run_seq2seq.py", line 27, in
from KGBART_model.tokenization_bart import MBartTokenizer, BartTokenizer
File "/workspace/KG-BART/KGBART/KGBART_model/init.py", line 2, in
from .tokenization_bart import BartTokenizer, BartTokenizerFast, MBartTokenizer
File "/workspace/KG-BART/KGBART/KGBART_model/tokenization_bart.py", line 19, in
from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
File "/workspace/KG-BART/KGBART/KGBART_model/tokenization_roberta.py", line 21, in
from tokenizers.processors import RobertaProcessing
ModuleNotFoundError: No module named 'tokenizers'

AttributeError: 'Namespace' object has no attribute 'commongend'

When I run reorder_src.py, I got this error AttributeError: 'Namespace' object has no attribute 'commongend'.

Run the model over a different KG

What are the steps to train the model for the CSKG dataset?

Inputs ignored during generation

I attempted to use the model for a separate task. With few, if any changes to the model, it seems that during generation, the model ignores all inputs to the decoder and predicts the most probable token each time. I would like to know if this is a known issue/if anyone else has encountered such an issue, if it is an issue with the code I wrote, or if this is a new issue that hasn't been seen before.

If anyone has tried working with this code and could give feedback, I would appreciate to hear from you, and thanks in advance.

KG-augmented decoding is not applied

Hello Ye, thanks for making the code public.

Not sure if I understand correctly, but from here it seems that the KG-augmented decoding layers are not applied during decoding. Could you let me know if there is anything I missed?

Also, could you check if there is any issue in the DecoderGATLayer? For example, from here, ex_entity may not be converted to embeddings before feeding into the exent_proj function.

Thanks again for the help.

Repurposing KGBART for storytelling

Hello. I'm interested in using KGBART to create a model capable of generating stories usin commonsense. I have a dataset with stories and the respective refered concepts. What I did was separate my dataset into train, test and dev sets and then created two files for each set: one containing a set of concepts per line (the src) and other with the correpsonding stories (the target).
But I think I need to preprare this data, right? I need to generate the knowledge graphs for the KGBART model?

I've tried to preprate the data as instructed (run reorder.py, conceptnet.py, train the entity and embedding using TranE), however when I trie to generate the one hop graphs by running entity_onehot.py it says that "commongen.train.index.txt" is missing.
What is this file? How should I generate it?

Thank you in advance.

Kg grounding，preparing data error

File "reorder_src.py", line 38, in
commongend = os.path.join(args.dataset_dir, args.commongend)
AttributeError: 'Namespace' object has no attribute 'commongend'

Error running Test

Hello,
i am interested on testing KG-BART but i am getting the following error:

can you advice?

missing file

Please point out where the file of "dataset/commongen.train.index.txt" is? When I ran entity_onehot.py, the program asks this file.
Thanks.

How to use the pretrained model to finetune for bigger documents ??

I see the max_seq_length and max_position is set to 32 and 64. Currently, when I was trying to do it with bigger size I was getting OOM even for batch size 1
I don't understand max_len_a , max_len_b and max_pred ?
Afaik these aren't specific to the BART paper

Thanks,
Steven

scripts n-n.py and train_transe_CommonGen.py are not included

Hello,

Thank you for your code.

I tried running

cd OpenKE
git clone -b OpenKE-PyTorch https://github.com/thunlp/OpenKE
cd benchmarks/CommonGen
python n-n.py
python train_transe_CommonGen.py

I can not find the scripts n-n.py or train_transe_CommonGen.py

Could you guide me here?