butsugiri / chainer-abcnn Goto Github PK

Re-implementation of Attention-Based Convolutional Neural Network (ABCNN) by Chainer

Python 16.15% Jupyter Notebook 83.85%

chainer-abcnn's Introduction

About

This repository contains re-implementation of Wenpeng Yin; Hinrich Schütze; Bing Xiang; Bowen Zhou. "ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs" TACL 2016 (PDF)

In order to assure that the implementation is correct, this repo reports the result for WikiQA task, which is the task used in the original paper.

@losyer gave me some very nice advice about the implementation (mainly about the model difference between the actual implementation by the author and the description provided on paper.)

WikiQA result

BCNN

BCNN HyperParameter

Number of BCNN layer: 1
Word Embedding: Pretrained word2vec vector from GNews
- Dimension is 300
Learning Rate: 0:08
Optimizer: AdaGrad
BatchSize: 20
Weight Decay: 0.0002

Hyperparameters are not tuned very well and the result is not as good as it should be. It seems that the final performance heavily depends upon the initial weight of the model.

Since the original paper does not report the hyperparameter for BCNN, I don't know whether my implementation is wrong, or it is parameter issue.

	MAP	MRR
SVM Dev	0.6307	0.6349
SVM Test	0.6165	0.6310
LR Dev	0.6307	0.6349
LR Test	0.6330	0.6480

The parameter of both SVM and LR are default of scikit-learn library.

ABCNN2

ABCNN2 HyperParameter

Number of layer: 2
Word Embedding: Pretrained word2vec vector from GNews
- Dimension is 300
Learning Rate: 0.05
BatchSize: 32
Optimizer: AdaGrad
Weight Decay: 0.0003

	MAP	MRR
Softmax Test	0.6614	0.6779
SVM Test	0.6301	0.6470

(Softmax means the direct prediction of the model, not relying on the external training of SVM/LR)

It seems that my implementation of (A)BCNN is significantly vulnerable to the initial value of paramters of conv. filter, final linear layer and the word embedding.

Here is the change in loss value of the same model (but different seed)

While the validation loss value decreases in one seed, the other one doesn't.

This issue very harmful to the actual performance of the model, as shown below.

The model with the seed value of decreasing validation loss, the model performs very good (although still worse than the score on the paper). However, the model with the other seed value performs poorly. The difference is 20 points in maximum, and this result is definitely undesirable.

ABCNN1 & ABCNN3

These models are implemented on this repo, but there is no experimental results.

chainer-abcnn's People

Contributors

Stargazers

Watchers

Forkers

kurtespinosa dhvfanny

chainer-abcnn's Issues

Question about vocab of dataprocessor.py

Dear Butsugiri,

Thank you for sharing your code. I just have a clarification about dataprocessor.vocab variable. After running the following lines:

data_processor = DataProcessor(args.data, args.vocab, args.test, args.max_length)
data_processor.prepare_dataset()
data_processor.compute_max_length()
train_data = data_processor.train_data
dev_data = data_processor.dev_data
test_data = data_processor.test_data

dataprocessor.vocab variable only has 2 entries and and hence this will be the input to the model creation.

cnn = ABCNN(n_vocab=len(vocab), embed_dim=embed_dim, input_channel=input_channel,
           output_channel=50, x1s_len=x1s_len, x2s_len=x2s_len, model_type=model_type, single_attention_mat=args.single_attention_mat)  # ABCNNはoutput = 50固定らしいが．
model = Classifier(cnn, lossfun=sigmoid_cross_entropy,
                     accfun=binary_accuracy)
if args.glove:
    cnn.load_glove_embeddings(args.glove_path, data_processor.vocab)
if args.word2vec:
    cnn.load_word2vec_embeddings(args.word2vec_path, data_processor.vocab)
if args.gpu >= 0:
    cuda.get_device(args.gpu).use()
    model.to_gpu()
cnn.set_pad_embedding_to_zero(data_processor.vocab)

Sorry, I haven't finished reading the whole code but I wonder at this point if that is the intention of that variable or it should have contained all the vocab in the dataset?

Cheers,
Kurt

Question about vocab variable in the dataprocessor.py file

Dear Butsugiri,

Thank you for sharing your code. I just have a clarification about dataprocessor.vocab variable. After running the following lines:

data_processor = DataProcessor(args.data, args.vocab, args.test, args.max_length)
data_processor.prepare_dataset()
data_processor.compute_max_length()
train_data = data_processor.train_data
dev_data = data_processor.dev_data
test_data = data_processor.test_data

dataprocessor.vocab variable only has 2 entries and and hence this will be the input to the model creation.

cnn = ABCNN(n_vocab=len(vocab), embed_dim=embed_dim, input_channel=input_channel,
           output_channel=50, x1s_len=x1s_len, x2s_len=x2s_len, model_type=model_type, single_attention_mat=args.single_attention_mat)  # ABCNNはoutput = 50固定らしいが．
model = Classifier(cnn, lossfun=sigmoid_cross_entropy,
                     accfun=binary_accuracy)
if args.glove:
    cnn.load_glove_embeddings(args.glove_path, data_processor.vocab)
if args.word2vec:
    cnn.load_word2vec_embeddings(args.word2vec_path, data_processor.vocab)
if args.gpu >= 0:
    cuda.get_device(args.gpu).use()
    model.to_gpu()
cnn.set_pad_embedding_to_zero(data_processor.vocab)

Sorry, I haven't finished reading the whole code but I wonder at this point if that is the intention of that variable or it should have contained all the vocab in the dataset?

Cheers,
Kurt

Clarification in the jsonify.py code

Dear Butsugiri,

Thank you for sharing your code. I have a question about the input dataset which I would need to jsonify. I download the dataset and used the respective data partitions, for example, WikiQA-test.tsv for test set which has a sample file entry below.

QuestionID Question DocumentID DocumentTitle SentenceID Sentence Label
Q0 HOW AFRICAN AMERICANS WERE IMMIGRATED TO THE US D0 African immigration to the United States D0-0 African immigration to the United States refers to immigrants to the United States who are or were nationals of Africa . 0

Now, I'm confused because in the jsonify code, the question would point to D0-0 which is the sentenceID. It seems that the question_id and the question were interchanged, am I right or did I miss out anything?

question_id = data[1]
....
question = data[-3]
answer = data[-2]
....
....
'question': question.lower().split(" "),
'answer': answer.lower().split(" "),

should have been the following?

question = data[1]
.....
question_id = data[-3]
answer = data[-2]
....
....
'question': question.lower().split(" "),
'answer': answer.lower().split(" "),

Cheers,
Kurt

Data

Could you upload the data?

Question: unexpecting results using MPR

Hi,
first of all than you for sharing a code, I was testing your code using Microsoft Research Paraphrase corpus that was tested in the original article as well. But I have a really poor result - accuracy only about 67% that is equal to the major voting strategy, as the majority class is in about 66% cases.

Have you tried your code on this dataset?

butsugiri / chainer-abcnn Goto Github PK

chainer-abcnn's Introduction

About

WikiQA result

BCNN

ABCNN2

ABCNN1 & ABCNN3

chainer-abcnn's People

Contributors

Stargazers

Watchers

Forkers

chainer-abcnn's Issues

Question about vocab of dataprocessor.py

Question about vocab variable in the dataprocessor.py file

Clarification in the jsonify.py code

Data

Question: unexpecting results using MPR

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent