yaushian / tree-transformer Goto Github PK

Implementation of the paper Tree Transformer

Python 100.00%

tree-transformer's Introduction

Tree Transformer

This is the official implementation of the paper Tree Transformer: Integrating Tree Structures into Self-Attention. If you use this code or our results in your research, we'd appreciate you cite our paper as following:

@article{Wang2019TreeTransformer,
  title={Tree Transformer: Integrating Tree Structures into Self-Attention},
  author={Yau-Shian Wang and Hung-Yi Lee and Yun-Nung Chen},
  journal={arXiv preprint arXiv:1909.06639},
  year={2019}
}

Dependencies

python3
pytorch 1.0

We use BERT tokenizer from PyTorch-Transformers to tokenize words. Please install PyTorch-Transformers following the instructions of the repository.

Training

For grammar induction training:
python3 main.py -train -model_dir [model_dir] -num_step 60000
The default setting achieves F1 of approximatedly 49.5 on WSJ test set. The training file 'data/train.txt' includes all WSJ data except 'WSJ_22 and WSJ_23'.

Evaluation

For grammar induction testing:
python3 main.py -test -model_dir [model_dir]
The code creates a result directory named model_dir. The result directory includes 'bracket.json' and 'tree.txt'. File 'bracket.json' contains the brackets of trees outputted from the model and they can be used for evaluating F1. The ground truth brackets of testing data can be obtained by using code of on-lstm. File 'tree.txt' contains the parse trees. The default testing file 'data/test.txt' contains the tests of wsj_23.

Acknowledgements

Our code is mainly revised from Annotated Transformer.
The code of BERT optimizer is taken from PyTorch-Transformers.

Contact

[email protected]

tree-transformer's People

Contributors

Stargazers

Watchers

tree-transformer's Issues

The group_prob & break_prob

in model.py,line 50
group_prob,break_prob = self.group_attn(x, mask, group_prob)may be
group_prob,break_prob = self.group_attn(x, mask, break_prob)?
cause the break_prob is neibor_attn in GroupAttention, which is the a in paper.

parse tree structure

Hi, thanks for your interesting work!

After reading the paper, I wonder why the parse tree may have three branches except the leaf nodes as shown in your paper Figure 3. For example, in your Figure 3, the root node have three subtrees.

Is it possible to do sentence classification using Tree Transformers?

question 1e-9？

Hi! I noticed that the the constant 1e-9 is used many times in your code(GroupAttention), why do you do this?
Thx

询问

您好博主，想问一下这个tree transformer有没有预训练好的比如hugging face社区可以直接下来拿来用的模型

Can't run without cuda

The code cannot be run without a working CUDA installation.
It would be beneficial to include the option, e.g. to be able to modify the architecture or debug in a computer without GPUs.

Share pre-trained model?

Do you have plans to share your pre-trained model?
I trained by myself using the WSJ data you provide, but my results, as explained in Issue #9 are not good. If you share your pre-trained model, I could test it.
Thanks!

Questions in paper

I found that the value of attention to oneself in the attention figure is 0, that is, the value of the diagonal part is 0, which means that one cannot attend to oneself. Will this affect performance? Is there any experimental verification? Thanks.

Code to reproduce Masked LM results

Could you please share the code to reproduce results in Sect 7.5 of your paper?

I am interested in using your model as a masked LM, to then generate sentences from it in a fashion similar to bert-gen. I implemented a similar procedure, using Tree-Transformer, with poor results. So my first step now is to see the predictions made by my trained model are reliable.

Why your WSJ-test and Ordered Neuron's WSJ-test have different number of instance?

Dear authors:

Hi, nice work!

I am trying to get the test data and compared yours and ON's here:
That file contains 3762 instances, while yours contains 2000+. Why there is a difference?

Thanks a lot.

Add LICENSE to repo

HI!

I am interested in using your code, but I noticed that the repo does not have a LICENSE.

Would it be possible to add a LICENSE to the repo? Thank you for the help!

For your reference, we are controlling random seeds using the following function:

def set_random_seed(seed=0):
    random.seed(seed)
    numpy.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)  # if you are using multi-GPU.
    torch.backends.cudnn.enabled = False 
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

If you may, could you please kindly share the random seeds or any specific settings aside from the default parameters for the Table 1 experiments for reproduction? Thank you!