dayu11 / differentially-private-deep-learning Goto Github PK

Hi,
I'd like to reproduce the experiments in Table 4.
And I could run Method RGP and RGP (N.P.) with this repository, but where is the proper code/code version for Method Full (N.P.) and DP-SGD?
I guess they use the original fairseq codebase or MC-BERT with opacus or pyvacy or sth like that.

cifar_train.py --eps 8 --delta 1e-5 --rank 16 --lr 0.5 --clipping 1 --batchsize 1000 --n_epoch 400 --width 10
Using the above parameters to train the cifar10 dataset under RGP, the accuracy is just over 40%. How to deal with it?
Test environment: win10, cuda11.1
I run it directly in pycharm IDE with the following settings:

parser.add_argument('--dataset', default='cifar10', type=str, help='dataset name')
parser.add_argument('--arch', default='resnet28', type=str, help='model name')  # resnet28
parser.add_argument('--resume', '-r', action='store_true', help='resume from checkpoint')
parser.add_argument('--sess', default='resnet', type=str, help='session name')
parser.add_argument('--weight_decay', default=1e-4, type=float, help='weight decay (default=1e-4)')
parser.add_argument('--batchsize', default=1000, type=int, help='batch size')
parser.add_argument('--n_epoch', default=400, type=int, help='total number of epochs')
parser.add_argument('--lr', default=0.5, type=float, help='base learning rate (default=0.4)')
parser.add_argument('--momentum', default=0.9, type=float, help='momentum coeeficient')

parser.add_argument('--eps', default=8, type=float, help='eps value')
parser.add_argument('--width', default=10, type=int, help='model width')
parser.add_argument('--delta', default=1e-5, type=float, help='delta value')
parser.add_argument('--rank', default=16, type=int, help='rank of reparameterization')
parser.add_argument('--clipping', default=1., type=float, help='clipping threshold')
parser.add_argument('--warmup_epoch', default=-1, type=int, help='num. of epochs for warmup')

Dear dayu11,

In your code, the parameter "warmup_epoch" is -1 by default, and it seems that you did not assign a new value to this parameter in your experiment.

However, if "warmup_epoch" is -1, it seems that at the train function:

This part of code will never be executed.

So, Is that correct or I am missing something?

Hi dayu11,
I'm appreciated that you open source the code of your brilliant work. I'm confused about the standard deviation of Gaussian noise in your implementation. In vanilla DP-SGD, the standard deviation equals to sigma, a.k.a. noise multiplier, times clip threshold.

I don't get the reason why you divide it by the batch size.
Is there some details I missed?

n = params[0].grad_batch.shape[0] AttributeError: 'Parameter' object has no attribute 'grad_batch'

Hi dayu11,
I'm a beginner in differential privacy. I'm grateful that you open source this outstanding work, my question may be silly but I wonder if there is a way to use DP independently?
I don’t know if I’ve expressed clear enough, I mean if I add noise to dataset or output(like DP-logits) instead of gradients, how can I achieve it? What are the key points in the code that I need to understand and apply to it?
I would be grateful if you could take the free time to answer this dumb question. Thanks for your help!!