thu-ml / ares Goto Github PK
View Code? Open in Web Editor NEWA Python library for adversarial machine learning focusing on benchmarking adversarial robustness.
Home Page: https://thu-ml-ares.rtfd.io
License: Apache License 2.0
A Python library for adversarial machine learning focusing on benchmarking adversarial robustness.
Home Page: https://thu-ml-ares.rtfd.io
License: Apache License 2.0
Thanks.
Hello, thanks for sharing.
Is there the adversarial training model for defense available in Keras or higher tensorflow in ImageNet and CIFAR-10?
Where can I refer to it?
Thanks & Regards!
Dear RealSafe team of thu-ml lab,
The realsafe code is in tensorflow version with 1.15.4, this means the CUDA version should be <=10.1.
In GeForce 3090 GPU infrasturctue environment, normally, the CUDA version should >= 11.1.
Do you have any tips to run on new GPU server, or do you recommend to run ares on 2080Ti or other older server?
Thanks & Regards!
Momo
I am trying to run the WideResNet_Trades model to run the benchmark using the benchmark.iteration_cli interface. However I am getting an error shown below.
Using TensorFlow backend.
WARNING:tensorflow:From /home/itachi/repos/ares/ares/benchmark/iteration_cli.py:100: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
WARNING:tensorflow:From /home/itachi/repos/ares/ares/benchmark/iteration_cli.py:105: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/itachi/repos/ares/ares/benchmark/iteration_cli.py:107: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-05-22 01:29:30.918086: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-05-22 01:29:30.946135: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2799925000 Hz
2021-05-22 01:29:30.946691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559ebfc9e4c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-05-22 01:29:30.946708: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-05-22 01:29:30.949124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-05-22 01:29:31.011989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-22 01:29:31.012647: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559ebfca0320 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-05-22 01:29:31.012678: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2021-05-22 01:29:31.012868: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-22 01:29:31.013139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2021-05-22 01:29:31.013309: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.013430: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.013538: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.013642: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.013730: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.013799: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /data/Softwares_Ubuntu/libs/cudnn-10.0-linux-x64-v7.4.2.24/cuda/lib64:/home/itachi/miniconda3/envs/cuda_toolkit/lib/
2021-05-22 01:29:31.016472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-22 01:29:31.016493: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1662] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-05-22 01:29:31.016511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-22 01:29:31.016521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2021-05-22 01:29:31.016529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
WARNING:tensorflow:From /home/itachi/repos/ares/ares/attack/utils.py:8: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/itachi/repos/ares/ares/dataset/utils.py:6: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use for ... in dataset:to iterate over a dataset. If using tf.estimator, return the Dataset object directly from your input function. As a last resort, you can use tf.compat.v1.data.make_one_shot_iterator(dataset).
Traceback (most recent call last):
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Variable_5
[[{{node Variable_5/read}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/itachi/repos/ares/ares/benchmark/iteration_cli.py", line 144, in <module>
rs = benchmark.run(dataset, logger)
File "/home/itachi/repos/ares/ares/benchmark/iteration.py", line 256, in run
return self._run(dataset, logger)
File "/home/itachi/repos/ares/ares/benchmark/iteration.py", line 113, in _run_basic
labels, dists = next(g)
File "/home/itachi/repos/ares/ares/attack/bim.py", line 115, in _batch_attack_generator
self._session.run(self.setup_xs, feed_dict={self.xs_ph: xs})
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/itachi/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Variable_5
[[node Variable_5/read (defined at /miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Original stack trace for 'Variable_5/read':
File "/miniconda3/envs/ares_benchmark/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/repos/ares/ares/benchmark/iteration_cli.py", line 138, in <module>
session, cw_n_points=args.cw_n_points, **kwargs)
File "/repos/ares/ares/benchmark/iteration.py", line 61, in __init__
self.attack = load_attack(attack_name, init_kwargs)
File "/repos/ares/ares/benchmark/utils.py", line 37, in load_attack
return attack_class(**kwargs)
File "/repos/ares/ares/attack/pgd.py", line 34, in __init__
self.rand_init_eps_var = tf.Variable(tf.zeros((self.batch_size,), dtype=self.model.x_dtype))
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 258, in __call__
return cls._variable_v1_call(*args, **kwargs)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 197, in <lambda>
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 2519, in default_variable_creator
shape=shape)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 262, in __call__
return super(VariableMetaclass, cls).__call__(*args, **kwargs)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1688, in __init__
shape=shape)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1872, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 203, in identity
ret = gen_array_ops.identity(input, name=name)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 4239, in identity
"Identity", input=input, name=name)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/miniconda3/envs/ares_benchmark/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
Loading tensorflow session...
Loading model...
Loading dataset...
Loading attack...
Configuring attack...
Running benchmark
Command:
python3 -m ares.benchmark.iteration_cli --method pgd --dataset cifar10 --count 100 --goal ut --distance-metric l_inf --offset 0 --output ./out_logs --batch-size 2 --iteration 10 example/cifar10/wideresnet_trades.py --logger
Below is my environment details:
OS: Ubuntu 20.04
Python: 3.7
Pytorch 1.5.0
Tensorflow: 1.15.4
Keras: 2.3.1
Does it have to do something with my setup or there's a bug? Please help me.
I see the overview says that ARES supports TensorFlow & PyTorch models with the same interface. but there is no direct function or interface for a trained Pytorch model to use. Does that mean user need to retrain their model with the decorator @pytorch_classifier_with_logits so that they could use the function in ARES to processing robustness testing?
尊敬的开发人员你们好,我想知道ARES是否无法直接使用我训练好的Pytorch模型进行他的鲁棒性测试?我看到我们的简介里说ARES以相同的接口支持tf和torch的模型。但是我仔细阅读我们的文档和代码后,并没有发现有类似最新版cleverhans的那种直接面向torch模型的接口。如果是我有所遗漏,请您指出我该如何操作。
Hello, I got an error when updating submodules.
Error log:
fatal: 引用不是一个树:c6e893728c3e7355e3bc689352011a55798b0e95
无法在子模组路径 'third_party/models' 中检出 'adc01cd76ae0d9d3b2e8dde3ec6bf4086f7da046'
无法在子模组路径 'third_party/trades' 中检出 'c6e893728c3e7355e3bc689352011a55798b0e95'
Is there any solutions?
Hi, I met an error of MPI when I run "python test/test_evolutionary.py":
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes
Below is the detail:
Traceback (most recent call last):
File "test/test_evolutionary.py", line 69, in
dists = next(g)
File "/home/***/realsafe/realsafe/attack/evolutionary.py", line 110, in _batch_attack_generator
args=[worker, xs_shm_file.name, xs_adv_shm_file.name, str(self.batch_size)])
File "mpi4py/MPI/Comm.pyx", line 1534, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes
[1028GQ-240:17362] *** Process received signal ***
[1028GQ-240:17362] Signal: Segmentation fault (11)
[1028GQ-240:17362] Signal code: Address not mapped (1)
[1028GQ-240:17362] Failing at address: 0x7f4fc075c980
Segmentation fault (core dumped)
Hello, we met an error when we run "python test_imagenet_models.py"
We run "python realsafe/example/imagenet/ens4_adv_inception_v3.py" and checked out that three chekpoint files have been downloaded successfully.
But the error occurred like:
NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint.
Hello, dear guys from thu-ml.
Thanks for your program.
I try to run test_imagenet_model.py for imagenet datasets with L∞ Attack.
The program runs very slowly.
I notice the requirement.txt set "tensorflow=1.15.4", not the "tensorflow-gpu=1.15.4".
I install with tensorflow-gpu 1.15.4 and make sure tensorflow.test.is_gpu_available() 's value TRUE.
I also notice:
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
My question is:
What others should I do when I run gpu program based on ImageNet.
I add adversarial attack code in:
https://github.com/thu-ml/realsafe/blob/master/realsafe/dataset/imagenet.py
def _load_image(filename, to_height, to_width, clip):
''' Load image into uint8 tensor from file. '''
img = Image.open(os.path.join(PATH_IMGS, filename))
if img.mode != 'RGB':
img = img.convert(mode='RGB')
if clip:
img = np.array(img)
height, width = img.shape[0], img.shape[1] # pylint: disable=E1136 # pylint/issues/3139
center = int(0.875 * min(height, width))
offset_height, offset_width = (height - center + 1) // 2, (width - center + 1) // 2
img = img[offset_height:offset_height+center, offset_width:offset_width+center, :]
img = Image.fromarray(img)
return tf.convert_to_tensor(np.array(img.resize((to_height, to_width))))
Before "if clip" and after " img = img.convert(mode='RGB')" is my modification, would this modification before tensor conversion and placeholder filling slow down the operation speed?
Thanks & Regards!
Momo
ares/ares/dataset/imagenet_dataset.py
Lines 8 to 23 in 306e35f
Hi, Can you please provide the meta_file used to load the dataset?
Can we apply the benchmark to our own model?
i want use other models to generate adversarial samples, but something wrong with it.can you tell me how to do it?
This is my code
import tensorflow as tf
import tensorflow_hub as hub
from realsafe.model.base import ClassifierWithLogits
from realsafe.utils import get_res_path
# MODEL_PATH = get_res_path('./cifar10/resnet56.ckpt')
def load(session):
model = VGGDEEP()
# model.load(MODEL_PATH, session)
return model
class VGGDEEP(ClassifierWithLogits):
def __init__(self):
ClassifierWithLogits.__init__(self,
x_min=0.0, x_max=1.0, x_shape=(32, 32, 3,), x_dtype=tf.float32,
y_dtype=tf.int32, n_class=10)
self.model = hub.Module("https://hub.tensorflow.google.cn/deepmind/ganeval-cifar10-convnet/1")
def _logits_and_labels(self, xs_ph):
logits = self.model(xs_ph)
predicts = tf.nn.softmax(logits)
predicted_labels = tf.argmax(predicts, 1, output_type=tf.int32)
return logits, predicted_labels
def load(self, model_path, session):
# var_list_pre = set(tf.global_variables())
# x_input = tf.placeholder(tf.float32, shape=(None,) + self.x_shape)
# self.model.inference(x_input, self.num_residual_blocks, reuse=tf.AUTO_REUSE)
# var_list_post = set(tf.global_variables())
# var_list = list(var_list_post - var_list_pre)
# if len(var_list) > 0:
# saver = tf.train.Saver(var_list=var_list)
# saver.restore(session, model_path)
pass
as you see ,i want to use this model(https://hub.tensorflow.google.cn/deepmind/ganeval-cifar10-convnet/1) to genernate adv samples.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.