Hi,

Are the scores from table 2 of the paper (Table 2: Performance on the Comparisons dataset.) from evaluating the model only on short splits (i.e. comp_ml_short, comp_not_short, comp_same_short)?

Or is it an average of both short and long splits? In the code is seems that it is only on the short.

Thanks

Hi,
this is not really an issue but more of a clarification question on the data (so please feel free to move it somewhere more appropriate). Can you please tell me which of the datasets contained in the folder testData/true was used for the negation set reported in your paper? I am guessing that subjv_long and subjv_short were included but were also comp_not_long and comp_not_short included or were the latter two part of the "same" set?

Thanks in advance for your help
Katerina

Hi again,

I am trying to run the main.py script in order to test the trained classifier models (BOWLogReg and InferSentLogReg) on another test set of my own. Unfortunately, I dont have the possibility to retrain the classifiers on the SNLI data due to hardware restrictions and that is why I would like to use the trained models but on a different test set. For this purpose I commented out the "loading training data" and "Training classifiers" parts of the main.py, in order to use the trained models. I have kept the lists of "names" and "classifiers" and I have adjusted the path TEST_DATA_PATH to point to the directory where my test set is located. When I run the script, it successfully starts and prints

[nltk_data] Downloading package punkt to /home/kkalouli/nltk_data...
[nltk_data] Package punkt is already up-to-date!
Vocab size : 100000
Running tests for tasks: ['test']

Running tests for InferSentLogReg...

but soon after, it throws the error

/home/kkalouli/Documents/libraries/ScrambleTests/classifier.py:154: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
Xbatch = Variable(devX[i:i + self.batch_size], volatile=True)
Traceback (most recent call last):
File "main.py", line 165, in
runAllTests(names, classifiers, model, tasks, outpaths, label2id)
File "main.py", line 125, in runAllTests
tF.runtests(name, classifier, model, tasks, outpaths, label2id)
File "/home/kkalouli/Documents/libraries/ScrambleTests/testFuncs.py", line 44, in runtests
labels, confs = predict(embedA, embedB, regressor)
File "/home/kkalouli/Documents/libraries/ScrambleTests/testFuncs.py", line 16, in predict
probs = reg.predict_proba(feats)
File "/home/kkalouli/Documents/libraries/ScrambleTests/classifier.py", line 156, in predict_proba
probas = self.model(Xbatch).data.cpu().numpy()
File "/home/kkalouli/Documents/virtEnv1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/kkalouli/Documents/virtEnv1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/kkalouli/Documents/virtEnv1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 484, in call
for hook in self._forward_pre_hooks.values():
File "/home/kkalouli/Documents/virtEnv1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 535, in getattr
type(self).name, name))
AttributeError: 'Softmax' object has no attribute '_forward_pre_hooks'

Do you have any ideas what the reason for this error could be and how to fix it? A quick google search was not very helpful on that aspect. Any other solving ideas are also welcome.

Thanks a lot in advance!