How to handle variant image sizes? Thanks about vit-pytorch HOT 3 CLOSED

In other words, if you pass in a 32x32 image (1 token), nothing will get learned in a transformer

from vit-pytorch.

@junyongyou Hi! I just released a new version of the library that will allow you to do this, provided when passing image_size in the constructor, you pass the maximum image size

import torch
from vit_pytorch import ViT

model = ViT(
	image_size = 512,
	patch_size = 32,
	num_classes = 1000,
	dim = 1024,
	depth = 6,
	heads = 8,
	mlp_dim = 2048,
	dropout = 0.1,
	emb_dropout = 0.1
)

x = torch.randn(1, 3, 256, 256)
y = torch.randn(1, 3, 512, 512)
model(x), model(y)

the other way to do this, is to embed your smaller image into the largest image size, then pass the appropriate mask to block out attention to the non-image regions

ex.

mask = torch.tensor([
  [1, 1, 0, 0]
  [1, 1, 0, 0]
  [0, 0, 0, 0]
  [0, 0, 0, 0]
]).bool()

x = F.pad(x, (0, 256, 0, 256), 0)
model(x, mask = mask)

from vit-pytorch.

Just throwing some caution though; you will need a sufficient number of patches for attention to work well, as attention relies on the comparison between the representative tokens of each patch

from vit-pytorch.