jinhaoduan / secmi Goto Github PK

Hi, I am doing some research on the privacy of diffusion models. Could I get the code you use to test GAN-Leaks？

Though you've explained it in issue6, I'd still like to double-check this detail cause I'm confused about the experiement result.

Referring to the official code for stable diffusion fine-tuning and the .sh example, there are two parameters for data transform during fine-tuning, namely args.center_crop and args.random_flip, representing centerCrop / randomCrop, no-Flip / randomFlip respectively.

If using the default parameter settings, it's actually a combination of randomCrop and no-Flip. Here's the code:

# Preprocessing the datasets.
train_transforms = transforms.Compose(
    [
        transforms.Resize(args.resolution, interpolation=transforms.InterpolationMode.BILINEAR),
        transforms.CenterCrop(args.resolution) if args.center_crop else transforms.RandomCrop(args.resolution),
        transforms.RandomHorizontalFlip() if args.random_flip else transforms.Lambda(lambda x: x),
        transforms.ToTensor(),
        transforms.Normalize([0.5], [0.5]),
    ]
)

I'd like to know which parameter combination you used？

I conducted experiments on various data-augmentation combination based on your methods (COCO dataset, 2500/2500 split, 150,000 steps, as described in your paper).

The experimental results align with yours only for randomCrop+ no-Flip (ASR/AUC: 0.8334/0.9105). And centerCrop +no-Flip yields higher results (ASR≈0.90). On the other hand, randomCrop+randomFlip results in much lower performance (ASR≈0.75).

So, did you use the combination of randomCrop and no-Flip?

Thank you for your time, and I look forward to your response!

Thank you for your work and for releasing the model weights

However, I obtained different results using your method on the official CIFAR-10 diffusion model ddpm-cifar10-google with the cifar10 official train/test split.

Could you please release the training/testing dataset corresponding to your pretrained ddpm model?

Thank you for your amazing work!

I'd like to inquire if there are evaluation codes about conditional diffusion model in this repository (Sec. 5.5 and Sec. 5.6 in paper) ?

Thank you for your inspiring work！

But I have one more question that arises from the closed issue:

You can assume we are in a shadow environment: training a shadow model with a known member/nonmember splitting and determining a threshold/NN according to this known splitting. After we get the threshold/NN, we could apply them to any real victim models (which we don't know the membership splitting).

We have some results to show that the threshold obtained from the shadow environment can be transferred to the real victim models:

Method Attack Target AUC ASR
SecMI_stat Shadow Model 0.881 0.811
SecMI_stat Victim Model - 0.804
SecMI_NNs Shadow Model 0.951 0.888
SecMI_NNs Victim Model - 0.893

Could you please describe how to construct the training data for the "shadow models"? When training a shadow model, do you assume that the adversary already knows the distribution of the dataset? (Or can obtain some of the training data?)

Hi,

Thanks for your great work! I have a question about the evaluation process. In my understanding, current implementations directly use the member/non-member labels to get the threshold for $SecMI_{stat}$ and for NN training $SecMI_{NNs}$, should not the attackers only have access to the data distribution $D$ instead of $D_M$ or $D_H$?

Hi, thank you for your paper and for sharing the code. May I ask some questions?

2. In Equation (13) in the paper, the $\tilde{x_t}$ is calculated many steps iteratively. Can we just calculate $\tilde{x_t}$ based on Equation (4) in one step?

Thanks.

Thank you for your great work and open-source code, which inspires me a lot.

During replication, there was a slight disparity between my results and yours (with ASR, AUC even higher on the Pokémon dataset than yours). So I want to know which differing settings led to my higher results.

My setting:
Pokmon train-test split: 416, 417.
Training steps: 15000, Batch size: 1, Gradient_accumulation_steps: 4, LR: 1e-5.
Without Crop and Flip. (Did you use crop and flip during training?)

My result
ASR 0.90, AUC 0.9391 with Prompt (higher than yours: 0.821, 0.891)

Trying to keep the settings consistent with the paper, but I still obtained different results. Looking forward to your response!

Dear Authors,

Thanks for your excellent work. I've noted the statistics for CIFAR10 in ./stats/cifar10.train.npz. Would it be feasible for you also to release the statistics for CIFAR100 and Tiny-ImageNet?

Thank you for your time and consideration.