Hi Dr Bargavj,

I looked at your constants.py file and find that the noise multipliers of rdp and gdp are pre-calculated given the specific epsilon.

"rdp_noise_multiplier = {
30: {0.01: 290, 0.05: 70, 0.1: 36, 0.5: 7.6, 1: 3.9, 5: 1.1, 10: 0.79, 50: 0.445, 100: 0.356, 500: 0.206, 1000: 0.157},
100: {0.01: 525, 0.05: 150, 0.1: 70, 0.5: 13.8, 1: 7, 5: 1.669, 10: 1.056, 50: 0.551, 100: 0.445, 500: 0.275, 1000: 0.219}
}"
"gdp_noise_multiplier = {
30: {0.01: 190, 0.05: 45, 0.1: 24, 0.5: 5.5, 1: 3, 5: 0.94, 10: 0.701, 50: 0.481, 100: 0.438},
100: {0.01: 350, 0.05: 82, 0.1: 44, 0.5: 10, 1: 5.4, 5: 1.43, 10: 0.955, 50: 0.564, 100: 0.498}
}"

Just wondering how did you get sigma given an epsilon value?

Thanks a lot.

FileNotFoundError: [Errno 2] No such file or directory: 'data/census/target_data.npz'

I have census_feature_desc.p ,census_feature.p ,census_feature.csv ,census_labels.p,census_labels.csv
but it need target_data.npz,holdout_data.npz,skewed_data.npz,skewed_2_data.npz

input:
python main.py census
--use_cpu=1
--skew_attribute=0
--skip_corr=1
--skew_outcome=3
--sensitive_outcome=3
--target_test_train_ratio=0.5
--target_data_size=50000
--candidate_size=10000
--target_model='nn'
--target_epochs=50
--target_l2_ratio=1e-6
--target_learning_rate=0.001
--target_batch_size=500
--target_clipping_threshold=4
--attribute=1
--run=1

output:
{'train_dataset': 'census', 'run': 1, 'use_cpu': 1, 'save_model': 0, 'save_data': 0, 'attribute': 1, 'candidate_size': 10000, 'skew_attribute': 0, 'skew_outcome': 3, 'sensitive_outcome': 3, 'banished_records': 0, 'skip_corr': 1, 'n_shadow': 5, 'target_data_size': 50000, 'target_test_train_ratio': 0.5, 'target_model': 'nn', 'target_learning_rate': 0.001, 'target_batch_size': 500, 'target_n_hidden': 256, 'target_epochs': 50, 'target_l2_ratio': 1e-06, 'target_clipping_threshold': 4.0, 'target_privacy': 'no_privacy', 'target_dp': 'gdp', 'target_epsilon': 0.5, 'target_delta': 1e-05, 'attack_model': 'nn', 'attack_learning_rate': 0.01, 'attack_batch_size': 100, 'attack_n_hidden': 64, 'attack_epochs': 100, 'attack_l2_ratio': 1e-06}
Traceback (most recent call last):
File "/Users/zhouyukai/PycharmProjects/EvaluatingDPML-master/improved_ai/main.py", line 332, in
run_experiment(args)
File "/Users/zhouyukai/PycharmProjects/EvaluatingDPML-master/improved_ai/main.py", line 135, in run_experiment
train_x, train_y, test_x, test_y = load_data('target_data.npz', args)
File "/Users/zhouyukai/PycharmProjects/EvaluatingDPML-master/core/data_util.py", line 204, in load_data
with np.load(DATA_PATH + data_name) as f:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/lib/npyio.py", line 390, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'data/census/target_data.npz'

when I run python attack.py cifar_100 --target_model=nn --target_l2_ratio=1e-4,

I met the error

Traceback (most recent call last):
  File "attack.py", line 595, in 
    run_experiment(args)
  File "attack.py", line 481, in run_experiment
    dataset = load_data('target_data.npz', args)
  File "attack.py", line 227, in load_data
    with np.load(DATA_PATH + data_name) as f:
  File "/home/billyluo/anaconda3/envs/tf14/lib/python3.7/site-packages/numpy/lib/npyio.py", line 428, in load
    fid = open(os_fspath(file), "rb")
FileNotFoundError: [Errno 2] No such file or directory: './data/target_data.npz'

How can I solve it?

The paper indicates that privacy leakage metric is always between 0 and 1, where the value of 0 indicates that there is no leakage.
When I run the code many times

$python3 main.py cifar_100 --target_model='nn' --target_l2_ratio=1e-4 --target_privacy='grad_pert' --target_dp='rdp' --target_epsilon=0.01 --target_epochs=100 --attack_epoch=100

I got the Advantage = -0.0009、0.0005、-0.0031、-0.0006、0.002
What does it mean when the privacy leakage metric is negative?
The smaller the privacy leakage metric, the more advantageous to the attack model?

I have run this code and getting all the graphs, such as accuracy loss and privacy leakage at various privacy budgets. I was wondering how can I generate the privacy leakage from the non-private model and plot it along with the perturbed results? I'd greatly appreciate if could help. Thanks.

Could you also share the parameters (learning rate, regularization parameter, etc) for purchase100?

And are you using the dataset from here ?

have I understood it correctly that you only implement Adversaries 1 and 2 of Yeom et al. (in yeom_membership_inference)? If so, was there a technical reason the colluding adversary (adversary 3) was not included in your analysis?

/core/attack.py imports 'plot_layer_outputs' from /core/utilities.py:

from core.utilities import plot_layer_outputs

This method does not exist and to the best of my knowlegde has never existed (in this repo).
Consequently running any command importing or using /core/attack.py raises an ImportError.

Is this method not supposed to be there? Where can I find it?

Hi Dr.Bargavj,

I am reading your paper "Evaluating Differentially Private Machine Learning in Practice" and have some questions as follows.

I noticed that you did experiments comparing effects on batch clipping and per-instance clipping, however, I did not find how to set parameters in your code to do this experiment. Does just setting "--target_batch_size=1" in evaluating_dpml.py file means per-instance clipping?

Many thanks if you can kindly reply to it.

Hi Dr Barga,

I read your paper 'Evaluating Differentially Private Machine Learning in Practice' and it is very interesting work.

when I run your code I find that in evaluating_dpml.py, it seems in line 22,
pred_y, membership, test_classes, classifier, aux = train_target_model(
dataset=dataset,
epochs=args.target_epochs,
batch_size=args.target_batch_size,
learning_rate=args.target_learning_rate,
clipping_threshold=args.target_clipping_threshold,
n_hidden=args.target_n_hidden,
l2_ratio=args.target_l2_ratio,
model=args.target_model,
privacy=args.target_privacy,
dp=args.target_dp,
epsilon=args.target_epsilon,
delta=args.target_delta,
save=args.save_model
),
should add 'args' to target_train_model()