archinetai / smart-pytorch Goto Github PK

Hi,
Thank you for the code.

• I don't understand why we try to compute the new noise (Line 50 to 54). Since algorithm 1 in the paper, didn't do the same.
• Which parameter is the momentum parameter?

Thanks.

Hello,

Thank you for providing the implementation but I seem to be running into the following issue

RuntimeError: One of the differentiated Tensors appears not to have been used in the graph. Set allow_unused=True if this is the desired behavior.

This happens at the following step:

loss = self.loss_fn(state_perturbed, state.detach())
# Compute noise gradient ∂loss/∂noise
noise_gradient, = torch.autograd.grad(loss, noise)

Any guidance on why this could be happening?

Hi, thanks for you implementation.
But i found this code seem imcompatible with DDP training, and errors would happen in line
# Get initial embeddings
embed = self.model.roberta.embeddings(input_ids)
in your https://github.com/archinetai/smart-pytorch#roberta-classification-example.