SVHN Checkpoint Result about robnets HOT 2 CLOSED

Hi,

I believe there are two problems here.

First, regarding the wrong checkpoint, sorry for this inconvenience. I have previously updated the correct version of the checkpoint on SVHN. You could download it from the link in README. What I want to stress here is that the mistake ONLY happens during the conversion from our private-trained models to pytorch-available models for the aim of publishing the code, more details described in #5 (comment), nothing wrong involved in the training or testing process using our private distributed framework. Therefore, the way you truncate the first 10 dimensions of the checkpoint actually gets the correct parameters.

Furthermore, regarding the results, the difference from the paper is because, when I was implementing the training code for this repo, I found the cosine-decay learning rate can help a lot to increase the final adversarial performance (where the numbers reported in the paper are using the step-decay learning rate scheme). I have tested it on CIFAR and SVHN. On CIFAR it works nice and I have updated it in the code, but for SVHN, it is hard to train. Additional tricks are required, i.e. you need to first train the model on clean data for several epochs then finetune it with the adversarial attack, and use a large batch size. And it helps the RobNet_free to work, yet unfortunately not for the RobNet_large. So the checkpoint of RobNet_free is trained using cosine-decay learning rate, and as you can see, it yields far much better results than those of the step-decay learning rate. We did not include this in the paper, since if we do, it is unfair for the comparison of other models. But for this public use, I believe the better performance will be more helpful.

(To help understand what I am saying, you can evaluate the result of RobNet_large using the same way you treat it with RobNet_free to get the correct performance and compare it with the paper. Sorry again for so much inconvenience introduced by the previous stupid mistake...)

from robnets.

Thanks.

from robnets.