Experiments with full convolutional layers instead of patch embedding？ about convmixer HOT 2 CLOSED

The models trained on CIFAR-10 that we discuss in the appendix actually do this, but that's because the patch size itself is 1.

For the ImageNet experiments, stride = 1 would result in a very large "internal resolution", which would make the model computationally infeasible.

And this wouldn't really result in a model like MobileNet -- our model is actually quite a bit more simple than MobileNet.

The fact that our model uses the patch embedding layer is a key aspect of our work -- achieving the highest possible accuracy was not our goal. Specifically, we intended to investigate the effect of using patches as the input representation, in the context of recent work on ViT and MLP-Mixer and their variants. If you remove the patch embedding stem, you would ultimately build a very different model.

There are definitely various ways you could tweak the architecture to increase performance (in terms of accuracy or throughput) while decreasing its simplicity, but the patch embedding stem itself would not be the first choice. :) This is definitely important future work, but it's secondary to the point of the paper.

from convmixer.

I'm going to close this issue, but feel free to reopen it or open a new issue if you have more questions.

from convmixer.