I have yet to run your code to check the results. However, it'd be great to have the tables available in the README.

I'd check the paper, but the link to your arxiv report leads to: https://arxiv.org/abs/TBD.

Modifications to run with python 3:
file-> open
iteritems -> items

- for line in file(vocab_file):
+ for line in open(vocab_file):

- for word, count in vbz.iteritems():
+ for word, count in vbz.items():

- for word, count in nn.iteritems():
+ for word, count in nn.items():

(BERT-Syntax) ant@..:~/.../bert-syntax$ python eval_bert.py | tee results/lgd_results_large.txt
using model: bert-large-uncased
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1248501532/1248501532 [18:31<00:00, 1123474.91B/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231508/231508 [00:00<00:00, 243357.01B/s]
Traceback (most recent call last):
File "eval_bert.py", line 128, in
eval_lgd()
File "eval_bert.py", line 81, in eval_lgd
for i,line in enumerate(open("lgd_dataset_with_is_are.tsv",encoding="utf8")):
FileNotFoundError: [Errno 2] No such file or directory: 'lgd_dataset_with_is_are.tsv'