gupta-abhay / pytorch-vit Goto Github PK

View Code? Open in Web Editor NEW

279.0 9.0 33.0 4.93 MB

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Home Page: https://arxiv.org/abs/2010.11929

License: MIT License

Python 100.00%

image-recognition transformers image-classification vit hybrid-vit vision-transformer

pytorch-vit's Introduction

Vision Transformers

Implementation of Vision Transformer in PyTorch, a new model to achieve SOTA in vision classification with using transformer style encoders. Associated blog article.

Credits to Phil Wang for the gif

Features

Citations

@article{dosovitskiy2020image,
  title={An image is worth 16x16 words: Transformers for image recognition at scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and others},
  journal={arXiv preprint arXiv:2010.11929},
  year={2020}
}

@article{xiao2021early,
  title={Early convolutions help transformers see better},
  author={Xiao, Tete and Singh, Mannat and Mintun, Eric and Darrell, Trevor and Doll{\'a}r, Piotr and Girshick, Ross},
  journal={arXiv preprint arXiv:2106.14881},
  year={2021}
}

@article{wang2021scaled,
  title={Scaled ReLU Matters for Training Vision Transformers},
  author={Wang, Pichao and Wang, Xue and Luo, Hao and Zhou, Jingkai and Zhou, Zhipeng and Wang, Fan and Li, Hao and Jin, Rong},
  journal={arXiv preprint arXiv:2109.03810},
  year={2021}
}

@article{zhai2021scaling,
  title={Scaling vision transformers},
  author={Zhai, Xiaohua and Kolesnikov, Alexander and Houlsby, Neil and Beyer, Lucas},
  journal={arXiv preprint arXiv:2106.04560},
  year={2021}
}

pytorch-vit's People

Contributors

Stargazers

Watchers

pytorch-vit's Issues

Using ViT

Hello Gupta!

Being new to vision tasks, can you share just a small snippet that can show how we can use pytorch-vit in downwards vision tasks like image retrieval etc. Thanx

Why only use the first patch? Thanks

I don't understand the line 74 of ViT.py:
x = self.to_cls_token(x[:, 0])
If the first dimension of x is batch, then the 2nd dimension 0 should be patch, as the dimension of x should be [batch, patch, feature]. Does it mean only the first patch is used? Could anybody help me on this? Thanks a lot.

layout for vit: NCHW or NHWC? bugpropagation

hello,
Thanks a lot for this very interesting work!

when you unroll the tensor you use unfold and flatten like this:

x = (x.unfold(2, self.patch_dim, self.patch_dim).
unfold(3, self.patch_dim, self.patch_dim).contiguous())
x = x.view(x.size(0), -1, self.flatten_dim)

but if x is in shape N,C,H,W, unrolling ends up with N,C,H//P,W//P,P,P and therefore flattening ends up mixing data from different channels. It means your "words" come from different blocks in space. It does not really matter for training your model with one specific size, but i think it will have hard time to transfer to a different size...

instead you could do like this:

self.flatten_dim_in = (patch_dim**2) * in_channels
...
x = (x.unfold(2, self.patch_dim, self.patch_dim)
         .unfold(3, self.patch_dim, self.patch_dim) .contiguous())
x = x.view(b,c,-1,self.patch_dim**2)
x = x.permute(0,2,3,1).contiguous()
x = x.view(x.size(0), -1, self.flatten_dim_in)

Just to make sure the data at the end is really what you expect: all the rgb pixels of one patch together, and not a mix of patches together.

Now i haven't tried your code yet so perhaps you have a different layout than N,C,H,W for images?

Pretrained models

Hi, will you release any pre-trained models?

Thank you

The reshape operation while using conv_patch_representation has some mistake.

https://github.com/gupta-abhay/ViT/blob/fcc17638d0f4d661af19128871345b01a800631c/vit/models/ViT.py#L99
I guess the self.flatten_dim in this line should be replaced with embedding_dim.

adjust_learning_rate can't import

In the train.py there the code include "from vit.utils import ( adjust_learning_rate)" but no adjust_learning_rate in the util.py

NameError: name 'lib' is not defined

Patch the image

How do you patch the image? any clues for the preprocessing and training step?

Working of FixedPositionalEncoding and LearnedPositionalEncoding?

I get the part where Image is split into P say 16x16 smaller image patches and then you have to Flatten the 3D patch to pass it into a Linear layer to get what they call Liner Projection. Can you please explain how the two types of Embeddings are working. Looked at your code too and looked like a maze to me. If you could just explain in Laymen's terms, I'll look at the code again and understand.
Thanks