Comments (4)
Our DP optimizer requires per_sample_gradient for every trainable parameter in the model (https://github.com/pytorch/opacus/blob/main/opacus/optimizers/optimizer.py#L283C17-L283C18). However, in your case, there is some part of the model not activated during (sometime) in the training, thus leading to a miss of per_sample_gradient. A potential fix might be freezing those parameters (setting param.requires_grad = False).
from opacus.
Thanks for your advise! I've tried to fix the problem in my code through your suggestion, but the code still can't work perfectly. Could you please tell me how did you fixed that problem through your method?
from opacus.
Thanks for the update. Could you elaborate the meaning of "the code still can't work perfectly"? Is there any new error popping up?
from opacus.
Here's the thing, I modified the code according to your suggestion using my own method. I modified the set_submodel method under the convnet class: after generating the submodel, I first froze all parameters and then enabled the parameters of the pre-set submodel. The following is the code for that part:
def set_submodel(self, ind, strategy=None):
self.ind = ind
assert ind <= 3
if strategy is None:
strategy = self.strategy
print(strategy, ind, self.dataset)
if strategy == 'progressive':
modules = []
for i in range(ind + 1):
modules.append(self.module_splits[i])
self.enc = nn.Sequential(*modules)
self.head = self.head_splits[ind]
for param in self.parameters():
param.requires_grad = False
for module in self.enc:
for param in module.parameters():
param.requires_grad = True
for param in self.head.parameters():
param.requires_grad = True
elif strategy == 'baseline':
modules = []
for i in range(len(self.module_splits)):
modules.append(self.module_splits[i])
self.enc = nn.Sequential(*modules)
self.head = self.classifier
for param in self.parameters():
param.requires_grad = False
for module in self.enc:
for param in module.parameters():
param.requires_grad = True
for param in self.head.parameters():
param.requires_grad = True
else:
raise NotImplementedError()
However, after rewriting it, the same error still occurs. I'm a bit confused about this. Could you please share the code you used to solve this problem?
from opacus.
Related Issues (20)
- UnsupportedModuleError: [IllegalModuleConfigurationError('Model needs to be in training mode')] HOT 1
- LLM finetuning with Opacus HOT 2
- Add context manager to toggle on/off privacy in training loop
- Some issue with loading model with 'weight' as opposed to 'pretrained=True' HOT 1
- Error: Trying to add hooks twice to the same model HOT 2
- OverflowError: cannot convert float infinity to integer HOT 2
- ModuleValidator.fix() causes layer gradients to be None. HOT 1
- Integrating Opacus for a custom pytorch model raises errors but works fine on its own HOT 2
- `BatchSplittingSampler` return wrong length HOT 5
- Error occurred when executing GroundingDinoSAMSegment (segment anything): Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_CUDA_mm)
- Why GDP and RDP give different result for the same config HOT 1
- Making a custom transformer architecture work with opacus HOT 3
- Spectral normalization in Opacus HOT 4
- BatchMemoryManager for training private GANs
- Wrapper references can be easily replaced, consider using properties instead HOT 3
- Error in DPOptimizer: Inconsistency between batch_first argument of PrivacyEngine and DPMultiheadAttention HOT 2
- Grad Sample Module: Use full backward hook to save activations and backprop values. HOT 1
- Microbatching Support HOT 4
- Training a simple transformer model with Opacus produces runtime error due to mismatch in dimensions HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opacus.