Comments (11)
Hi,
The PatchShuffle class is doing two things in sequence:
- create the mask, in which the cnn output here is only help to specify the dimensions.
- use the mask to mask out the input.
You can of course implement these two things separately with two classes or functions. I implemented in this way only for convinent. And it is different with the official implementation since when I wrote the code, the official one was not yet released.
And it is also very straightforward to understand which patch comes from which region of the image. Say your input is 224x224 image, and patch size is 14, then you will get a 16x16 grid of patches from the conv and each patch on this grid is from a 14x14 region from the original image without overlapping.
from mae.
Hi, thank you for sharing the code.
why did not you use sine-cosine positional embedding as it is mentioned in the paper?
from mae.
I don't find where they mention of using sin-cos positional embedding in the paper. Actually, the original ViT paper clearly mentioned that a "learned" positional encoding is added after patchfication. Also for images, it is not necessary to use the sin-cos positional encoding since there is no extrapolation beyond the trained length. Could you point out where you read it?
from mae.
Sure, in the paper https://arxiv.org/abs/2111.06377, on page 11, first paragraph.
from mae.
ah, I see. Thanks for the reference. I didn't pay much attention to this detail. But, as I said, I don't think it will make a large difference to the result. Feel free to experiment with that.
from mae.
Also, I just checked their official code and they don't even follow this detail. The code uses the ViT model from timm which follows the details in the ViT paper with learned positional encoding.
from mae.
https://github.com/facebookresearch/mae/blob/main/models_mae.py
You can see that they utilized the frozen positional embedding using the sine-cosine approach.
from mae.
Ah, thanks for the correction. I had looked at a wrong file. Then I don't know why they don't like to follow the ViT architecture precisely.
from mae.
Based on what I studied, we do not have specific rules for choosing the positional embedding. However, I want to try the sine-cosine approach, and see the result. If I test it, I will inform you.
I want to be sure that this implementation considers other details mentioned in the paper. I checked it, however, I want to be sure.
from mae.
Oh, I don't think I followed all the details from the paper precisely. As in the readme, the purpose of this code is only to verify the idea of mae, not a replicate. For example, I think I didn't implement the normalization for reconstruction loss. There could be more details that I missed.
from mae.
Related Issues (20)
- What are the rules for setting the parameters of vit-tiny's decoder? HOT 1
- Fine-tuning and linear evaluation HOT 4
- What are the versions you worked with, keep getting errors HOT 2
- Need help for reconstracting your experiments HOT 2
- how to make the masked patches not random HOT 2
- 最后你得重建图像是怎么来的? HOT 2
- Suspected minor bug HOT 1
- resource requirement HOT 2
- Ask some questions HOT 8
- tqdm is required package HOT 1
- train_classifier HOT 1
- 权重读取后重建图像不理想
- Expected dtype int64 for index HOT 3
- Could you provide a one-dimensional MAE implementation in pytorch? HOT 1
- Why do the masks' positions on the visualized images are same? HOT 4
- The training loss is decreasing while the validation loss is increasing HOT 4
- Unable to reconstruct a distinguishable image HOT 5
- extracting embeddings HOT 1
- Turn off masking HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mae.