michaelsdr / momentumnet Goto Github PK

• Flake 8
• Tests
• Make site

Is there any code for the experiment on figure 9 of the paper?
I want to compare three models in Figure 9.

In this example setting gamma=0.0 lead to mnet1 and net having two different outputs.

However, everything is fine with resnet18.

(it seems that it apply the residual connection of ResBlock after TransformMemory)

（1）不明白图一和论文的关系。（2）图一左右两边想表达什么意思。右边能表达更复杂的动力学行为？

I found that in the code:
v_{n+1} = gamma * v_n + (1-gamma) * f

However, in the document, it is written that:
v_{n+1} = (1-gamma) * v_n + gamma * f

The following code leads to an error using torch version 1.10 but not 1.9

import torch
from torch import nn
from momentumnet import MomentumNet

init_function = nn.Tanh()
functions = [nn.Linear(3, 3), ]

net = MomentumNet(functions, init_function=init_function, init_speed=True, gamma=0.9,
                                  use_backprop=False)

x = torch.randn(1, 3)

output = net(x).sum()
output.backward()

Changing init_function to something with parameters like

import torch
from torch import nn
from momentumnet import MomentumNet

init_function = nn.Linear(3, 3)
functions = [nn.Linear(3, 3), ]

net = MomentumNet(functions, init_function=init_function, init_speed=True, gamma=0.9,
                                  use_backprop=False)

x = torch.randn(1, 3)

output = net(x).sum()
output.backward()

works fine.

No idea what causes this.

Hi thanks so much for your work!

Is it possible to add a method for computing the inverse of a layer's activations? Both the iResNet and iRevNet repos have this method. Adding this would really help me experiment with your model more.

Thanks!

Dear Michael,

Thank you for making your work publicly available. I was wondering your opinion on including momentum in DenseNets. More specifically, do you think that adding momentum in each layers that are connected to every layer in a neural model would yield similar gains in memory?

Thanks for your interesting work!

The Reformer uses RevNet in a clever way. They double the dimension of x such that for x1,x2=split(x) both x1 and x2 have the same dimension as the original x. This gives their invertible architecture the "same parameters" as the initial architecture. Let's call this ReformerRevNet.

Question 0. In Table 2, RevNet differs to MomentumNet only in the row "same parameters". I don't see why ReformerRevNet and MomNet would be different in Table 2?

Question 1. Is there any reason this ReformerRevNet baseline was not included?

Apologies for any misunderstanding.

I tested the memory consumption of resnet 101 and the corresponding momentum version using the scripts below. However, difference in memory consumption between these 2 models is only a few hundreds MBs.

Pytorch version: 1.9
CUDA toolkit: 11.1
GPU: GTX 1080 Ti
Python 3.7
Ubuntu 18.04

from momentumnet import transform_to_momentumnet
from torchvision.models import resnet101

resnet = resnet101(pretrained=True).to(0)
mresnet101 = transform_to_momentumnet(resnet, gamma=0.99998, use_backprop=False)
#mresnet101 = resnet101(pretrained=True).to(0)
optim = torch.optim.Adam(mresnet101.parameters(), lr=1e-4)

for i in range(100):
    x = torch.randn(70, 3, 224, 224).cuda()
    y = mresnet101(x)
    y = (y**2).mean()
    optim.zero_grad()
    y.backward()
    optim.step()