megvii-model / weightnet Goto Github PK

Hi,
Thank you for this great work and code!
When I tried to run the code with multi-gpus, I got an error in the following line.

And the error log is as follows:

loss = dist.all_reduce_sum(loss, "train_loss") / dist.get_world_size()
File "/opt/conda/lib/python3.7/site-packages/megengine/distributed/functional.py", line 238, in all_reduce_sum
assert _group_check(key, nr_ranks), "key, nr_ranks should be set at the same time"
AssertionError: key, nr_ranks should be set at the same time

I'm new to using MegEngine and could not find any related issues by googling.
Please let me know how to resolve this error.
Thank you!

Hi,

I am trying to apply WeightNet to my own project and have a question about the second grouped fully connected layer of WeightNet: why is the bias set to false? Is there any reason behind this?

Hi,

I have a question about applying WeightNet to residual blocks with upsampling/downsampling. In this case, the skip path also has a 1x1 convolution. Should this conv layer also be replaced by WeightNet? Or it is better to keep it as standard convolution?

class WeightNet_DW(M.Module):
r""" Here we show a grouping manner when we apply WeightNet to a depthwise convolution.

The grouped fc layer directly generates the convolutional kernel, has fewer parameters while achieving comparable results.
This layer has M/G*inp inputs, inp groups and inp*ksize*ksize outputs.

"""
def __init__(self, inp, ksize, stride):
    super().__init__()

    self.M = 2
    self.G = 2

    self.pad = ksize // 2
    inp_gap = max(16, inp//16)
    self.inp = inp
    self.ksize = ksize
    self.stride = stride

    self.wn_fc1 = M.Conv2d(inp_gap, self.M//self.G*inp, 1, 1, 0, groups=1, bias=True)
    self.sigmoid = M.Sigmoid()
    self.wn_fc2 = M.Conv2d(self.M//self.G*inp, inp*ksize*ksize, 1, 1, 0, groups=inp, bias=False)

# x_gap是经过AGP的值
def forward(self, x, x_gap):
    x_w = self.wn_fc1(x_gap)
    x_w = self.sigmoid(x_w)
    x_w = self.wn_fc2(x_w)

    x = x.reshape(1, -1, x.shape[2], x.shape[3])
    x_w = x_w.reshape(-1, 1, 1, self.ksize, self.ksize)
    x = F.conv2d(x, weight=x_w, stride=self.stride, padding=self.pad, groups=x_w.shape[0])
    x = x.reshape(-1, self.inp, x.shape[2], x.shape[3])
    return x

When I convert it to the Pytorch version, I have the following problems. Can you help me solve them？thanks
error:
x_gap = x.mean(axis=2,keepdims=True).mean(axis=3,keepdims=True)
TypeError: mean() received an invalid combination of arguments - got (axis=int, keepdims=bool, ), but expected one of:

• ()
• (torch.dtype dtype)
• (tuple of ints dim, torch.dtype dtype)
  didn't match because some of the keywords were incorrect: axis, keepdims
• (tuple of ints dim, bool keepdim, torch.dtype dtype)
• (tuple of ints dim, bool keepdim)
  didn't match because some of the keywords were incorrect: axis, keepdims

Thanks for this outstanding work, I'm wondering why the weightnet.py use 5D tensors as weights for F.conv2d?
Does this really work for F.conv2d?

Actually this will raise errors for PyTorch, so is there a different behavior using MegEngine?
e.g. the code here.

Looking forward to your reply.

Though warming up is adopted, it also can't convergence.