what is the attributes in the large-kernel CNN about how-do-vits-work HOT 2 OPEN

Thank you for your support and insightful question!

In our observation, the attributes of Conv depend primarily on the architecture or the group size (e.g., depthwise-separable Conv) rather than the kernel size. For example, some Conv blocks at the end of stages of ConvNeXt behave like a low-pass filter in terms of Fourier analysis:

This figure provides ∆ log amplitude of ConvNeXt at high-frequency. ConvNext layers at the end of stages reduce high-frequency information, but ResNet layers do not. We leave a detailed investigation for future work.

∆ log amplitude of ResNet-50 at high-frequency

Cf. MSAs generally behave like low-pass filters and reduce feature map variance. Convs, on the contrary, behave like high-pass filters and increase the feature map variance. In terms of feature map variance, all ConvNeXt blocks diversify feature maps (i.e., they always increase the variance).

from how-do-vits-work.

[1] A ConvNet for the 2020s.
[2] Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs.

from how-do-vits-work.