rfeinman / detecting-adversarial-samples Goto Github PK

The code that creates the detector (linear regression classifier):

def train_lr(densities_pos, densities_neg, uncerts_pos, uncerts_neg):
    # [...data preparing code left out...]
    
    lr = LogisticRegressionCV(n_jobs=-1).fit(values, labels)
    return values, labels, lr

At the end the detector is evaluated on the data it was trained on (line 159 uses values which is the training data returned by train_lr):

Evaluating on training data leads to a biased result. Did I miss anything?

In the abstract of the paper it says "The result is a method for implicit adversarial detection that is oblivious to the attack algorithm. "

But the final detector is trained on adversarial examples generated by specific attack algorithms (see lines 151 and 153):

Following the README, trying to execute:
python craft_adv_samples.py -d=mnist -a=fgsm
got me the following error:

Traceback (most recent call last):
   File "craft_adv_samples.py", line 124, in <module>
    main(args)
  File "craft_adv_samples.py", line 99, in main
    args.batch_size)
  File "craft_adv_samples.py", line 40, in craft_one_type
    clip_max=1., batch_size=batch_size
  File "/home/user/detecting-adversarial-samples/src/attacks.py", line 185, in fast_gradient_sign_method
    [X, Y], args={'batch_size': batch_size}
  File "/home/user/detecting-adversarial-samples/scripts/src/cleverhans/cleverhans/utils_tf.py", line 350, in batch_eval
(...)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'dropout_1/keras_learning_phase' with dtype bool
         [[Node: dropout_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

It seems that the cleverhans function batch_eval needs the dropout layers to be explicitely turned off.
For the specific case of the tutorial, modifying src/attacks.py in line 185 as follows:

    X_adv, = batch_eval(
        sess, [x, y], [adv_x],
-       [X, Y], args={'batch_size': batch_size}
+       [X, Y], args={'batch_size': batch_size}, feed={'dropout_1/keras_learning_phase:0': False}
        )

solves the problem.

I suppose that other models won't necessarily have that exact name for their dropout layers (haven't tested them) so I can't really vouch for this solution as an always working patch.

I am trying to generate the adversarial images on SVHN database but getting following execution error. Can you please help or it would be helpful if you can provide me the adversarial images from SVHN database (if possible)
Traceback (most recent call last):
File "train_model.py", line 5, in
from detect.util import get_data, get_model
File "/home/iiitd/caffe/python/detect.py", line 24, in
import caffe
File "/home/iiitd/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
File "/home/iiitd/caffe/python/caffe/pycaffe.py", line 13, in
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
ImportError: dynamic module does not define module export function (PyInit__caffe)

I was trying to reproduce the results of the paper. Can you suggest how do I get the results for the CW attack ?