meghdadk / scrub Goto Github PK

i have read a part of the mia code,but I found that the logistic regression model in the code seems to be used without training for member inference attacks. Is this correct?

First of all, Thanks for the good work.
While trying out the code I found that some of the hyperparameters are not listed in the paper or the hyperparameters detailed in the paper are not used in the experiments.

The hyperparameters for the "Original" and "Retrain" were not listed in the paper(or I couldn't find them). And for the detailed hyperparameters of "EUK" are mentioned as "same setting as Retrain baseline". Based on the main_merged.py I assumed that detailed hyperparameters of "Retrain" are [epochs 26, fixed learning rate 0.01, weight_decay 5e-4, momentum 0.9]. But In the MIA_experiments.ipynb the "EUK" baseline uses learning rate decay every 10, 15, and 20 epochs and uses 0 for the optimizer's weight_decay and the momentum.

lastly, "Finetune", "Negrad", "CFK", and "EUK" should use weight_decay of 5e-4 but it is implemented to use 0.

This got me confused. Can you provide the exact value for the hyperparameters? (including epochs, learning rate, weight_decay, momentum, lr decay)

Thank You! If I missed something let me know

First of all, Thank you for the great work!
While trying out your work, I discovered that your repository does not include the LiRA MIA metric. May I ask for the LiRA MIA code?

Also, I've 2 questions related to LiRA MIA.
Each data example is seen exactly half of the time in the train and half of the time in the test when creating the shadow original models. Then when choosing to create 256 unlearning models shouldn't the number of shadow original models be 512?

If my understanding is correct, in LiRA MIA evaluation the Attacker needs to perform 2,560,000 unlearning to create 2,560,000 shadow unlearned models. And after, the Attacker selects 256 shadow unlearning models per example to perform the LiRA Attack. Is there a reason for creating 10k unlearned models per shadow original model instead of one? (I've read that the strength of the attacker is heavily dependent on the number of shadow original/unlearned models. But isn't the suggested MIA create Gaussian Distribution after the Attacker selects 256 models?)

All replies pointing out my understanding are welcome!
Thank you!!

The implementation of EU/CF updates the 'first K' layers of the network instead of the 'final K' as proposed originally in the associated paper. Moreover, the numbers reported are for the weakest version, where only the final layer is unlearnt.

We request the authors to fix this, or mention the mistake explicitly on the repository to prevent further misreporting of our work.