Comments (7)
as my understanding, the attention weights matrix is 17*17, CLS token + frame 1-16.
if we want to visualize the CLS token, we need to take out the first row and columns 1-16 of the matrix. it represents the attention scores between CLS token and frames 1-16.
We align them to raw images 1-16, each frame will have only one attention score.
In your paper, Fig.7, a single frame is, however, corresponding to 16 different values and has 16 different highlights.
I am wondering if we only visualize the CLS token, how could we get 16 different values on a single frame? Shouldn't one frame have only one value that represents the similarity of the frame to CLS token?
It would be great if you could give me some help.
Thanks
Best
from slowfast.
Hi @ziyuleoliu,
The visualization we suggest shows which frames contributed more to the classification and not which part in the image (pixel-wise) was most relevant.
See issue #17 on how to implement this visualization.
from slowfast.
hey
thanks for your reply :)
i'm trying to use the same framework, but vanilla transformer instead of Longformer. The cls token in my case is a [17,17] matrix. (1 CLS token + 16 frames). If I try to visualize the CLS token and take the first row of the matrix. I can only get an array of 17. It can show the contribution of each frame to CLS token. But each frame has only 1 value. But in your paper figure 7, each frame has 16 different values.
I'm wondering what makes this difference. Is it because of Longformer?
Your framework inspires me a lot, I would love to ask if it's possible to visualize the grad-CAM of the temporal encoder?
Normally if we want to visualize the VIT we have to resize the token [bs, 196, dim] to [bs, 14,14,dim]. but in this framework, temporal encoder takes directly the feature vectores as tokens eg. res50 (bs, 16, 768). Can we still visualize the gradcam of temporal encoder and put the heatmap on org pics?
Thanks again for your time.
Best wishes from noob Leo
from slowfast.
hey @bomri
i would be appreciate if you could give me a bit more help. i stuck here since weeks and get really confused....
Best
from slowfast.
Hi @ziyuleoliu ,
This figure's visualization uses the weights correlated to the CLS token. If you are not using the Longformer, how do you consider the temporal information? we use the "temporal" weight per frame in the clip to illustrate this.
We haven't tried grad-CAM , for temporal heatmaps, we use what is shown in the paper, for spatial it can probably work similarly to other usages of grad-Cam.
from slowfast.
hey @bomri
thanks for your reply.
I use a vanilla transformer (VIT) instead of Longformer as Temporal Encoder to model the temporal information.
The temporal attention weights is a 17*17 matrix:
If I have the right understanding, I should only take the first row and column [1:17](frame1 to 16). It represents CLS as Query and frame 1 to 16 as Key, so that i can get which frame has the biggest contribution for CLS token e.g., the classification result, right? Or should i take the first column and row [1:17]?
But what confused me is one frame has only one weight value correlated to CLS token, as it shown in the attachment.
But in your figure, a single frame will have different values.
I'm worried that i have an incorrect understanding and took the wrong weights.
I appreciate your time and it would be great if you can give me a bit more help.
Thanks again.
LEO
from slowfast.
If I have the right understanding, I should only take the first row and column [1:17](frame1 to 16). It represents CLS as Query and frame 1 to 16 as Key, so that i can get which frame has the biggest contribution for CLS token e.g., the classification result, right?
Right
Or should i take the first column and row [1:17]?
It depends on your implementation and I don't remember if we used row or col in our setup.
But what confused me is one frame has only one weight value correlated to CLS token, as it shown in the attachment.
The image you added looks good, seems like the info in the start and end are the most important
But in your figure, a single frame will have different values.
We also have one value per frame (same as in your case). We get one value per sec in a 10 sec video, sampled with 25fps, so 250 values which we plot in the top, but for the video images we sub-sample only for making it easier for visualizing the video with 16 frames.
from slowfast.
Related Issues (17)
- how to prepare kinetics400 dataset to run your code? HOT 1
- Thermal diagram of key frame HOT 1
- Unable to load weights of the 'VIT_B_VTN' model HOT 1
- VTN not converging on UCF101 HOT 2
- Error with Detectron2 when using run_net.py HOT 1
- VTN input example HOT 4
- How to use Resnet as backbone? HOT 2
- How to visualize the cls token in VTN ? HOT 2
- size of checkpoint file HOT 1
- Have you preprocessed the kinetics dataset? HOT 1
- Where is the code?
- VTN attention window HOT 2
- Position IDs in VTN HOT 1
- Reproducing top 1 accuracy in Figure 4 HOT 6
- How to reproduce result in live camera feed without saving as video HOT 2
- Model weights for ViT-B-VTN (1 layer) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from slowfast.