cybertronai / autograd-hacks Goto Github PK

Hello, is there an implementation with multi GPU compatibility for AutoGrad Hacks library ?

My model outputs a vector, but I actually only need the grad for one scalar in that vector. Is there a way I can get this to work?

Getting this warning:

UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.

Propose to substitute the following line:

with:
A = torch.nn.functional.unfold(A, layer.kernel_size, dilation=layer.dilation, padding=layer.padding, stride=layer.stride)

Hi Yaroslav,
First of all, I want to say your tricks here are really neat and have helped me a lot!

I'm currently trying to compute the per-example gradient over two backprops in a typical Hessian vector product calculation, e.g.

output = model(data)
loss = loss_fn(output, targets)
grad = torch.autograd.grad(loss, model.parameters(), create_graph=True)
grad = flat_grad(grad)
grad_p = (grad * p).sum()
hessian_p= torch.autograd.grad(grad_p , actor.parameters())

Is it possible to apply compute_grad1 to this procedure to get the per-example Hessian vector product? Naively applying it fails because currently compute_grad1 only supports a single backprop.

Thanks!
Jack

Hi! My question is about computing gradients for each sample in the batch. I reproduced your example with simple neural net, which works exactly fine. However, when using ResNet50 (which has batchnorm, for example) the code produces the following error:
AttributeError: 'Parameter' object has no attribute 'grad1'

My model outputs a vector, but I actually only need the grad for one scalar in that vector. Is there a way I can get this to work?

autograd_hacks.compute_grad1() TypeError: compute_grad1() missing 1 required positional argument: 'model'

This assertion failed at some point during training...

hack_grads = torch.cat([param.grad1.flatten(start_dim=1) for param in model.parameters()
                                       if hasattr(param, 'grad1')], dim=1)

grads = torch.cat([param.grad.flatten() for param in model.parameters() if hasattr(param, 'grad1')], dim=0)
                
assert torch.allclose(torch.mean(hack_grads, dim=0), grads)

I suspect supporting Conv1d and Conv3d is straight forward, but einsum() probably needs to change? Does anyone know the correct values here for the computation string?

Hi,
First, thanks a lot for this library!

Recently I found a bug in this line of your code:

The bug makes the code not compatible with non-default padding and stride in Conv2d.
Hence, this should be corrected to
A = torch.nn.functional.unfold(A, layer.kernel_size,padding=layer.padding,stride=layer.stride)

Best regards,
Haoxiang

In a nutshell, my code looks like this:

autograd_hacks.add_hooks(model)
all_params = []
for i in range(20):
      epoch_params = []
      train_loss = training_function(model, data, lr)
      autograd_hacks.compute_grad1(model)
      for name, params in model.named_parameters():
            sample_grads = params.grad1.clone().cpu().detach().numpy()
            epoch_params.append(sample_grads)
      all_params.append(epoch_params)
      autograd_hacks.disable_hooks(model)

all_params should contain different values as the gradients are changing every epoch, but it outputs always the same array. I tried using remove_hooks and clear_backprop but either it gave me errors or it does nothing. The training function has the usual loss, step, etc. I'd imagine the solution to this is easy. If it is not, I can write a minimal reproducible example.

Module backward hooks may get deprecated eventually, can replace them with tensor hook as follows.

def tensor_hook_adder(module, input, output):
    def tensor_backwards(grad):
        setattr(module, 'backprops', grad)
    output.register_hook(tensor_backwards)

layer.register_forward_hook(tensor_hook_adder)