jhyuklee / sparc Goto Github PK

Hi,

Thank you for the great project.
I followed the instructions to train the bert-large model on SQUAD, and encoded my dataset both with the model you provided (denspi_sparc) and my trained model. However, the results with my model look significantly worse, though not completely off. For example, a single-word search returns 55 exact word matches in search results with denspi-sparc, but only 27 with my trained model, and the best results are not found. I had to make one substantive code change: replaced run_natkb.py in

( return ["python", "run_natkb.py" ) with train.py, as run_natkb.py is not provided.
The results of my model training are:

│08/19/2020 15:57:05 - INFO - post - num vecs=45059736, num_words=1783576, nvpw=25.2637
08/19/2020 15:57:08 - INFO - main - [Validation] loss: 10.361, b'{"exact_match": 76.27246925260171, "f1": 84.389824154709}\n'

which are similar to your posted results:

04/28/2020 06:32:59 - INFO - post - num vecs=45059736, num_words=1783576, nvpw=25.2637
04/28/2020 06:33:01 - INFO - main - [Validation] loss: 8.700, b'{"exact_match": 75.10879848628193, "f1": 83.42143097917004}\n'

The sparse weights for input_examples.txt also look similar to yours.

Any help would be appreciated.