Comments (2)
A follow-up question I have is, when the training input has 16 frames, isn't the long former == normal transformer? What's the meaning of setting window size here? No matter the size is 18 or 32, it's not smaller than 16. Then the temporal attention encoder is O(n^2) transformer right? What's the benefit of using Longformer then? Please correct me I misunderstood something.
Thank you.
from slowfast.
Hi @YunYunY,
Thank you for your comments.
To reproduce the results you should use the config and the code in this repository - ATTENTION_WINDOW: [18, 18, 18]
. The misalignment in window size between the paper and the code is that different implementations of the Longformer model handle this value slightly different in terms of padding the input sequence and sliding windows size, and we moved from one implementation to the other [1] to [2].
Regarding the second comment, the sliding window is effective when evaluating full videos and the input sequence is much longer than 16 (250 frames for Kinetics-400). In this case, the Longformer attends locally using the sliding window and globally using the CLS token.
from slowfast.
Related Issues (17)
- how to prepare kinetics400 dataset to run your code? HOT 1
- Thermal diagram of key frame HOT 1
- Unable to load weights of the 'VIT_B_VTN' model HOT 1
- VTN not converging on UCF101 HOT 2
- Error with Detectron2 when using run_net.py HOT 1
- VTN input example HOT 4
- How to use Resnet as backbone? HOT 2
- How to visualize the cls token in VTN ? HOT 2
- Visualization: how effective is a PART of frames on decision making HOT 7
- size of checkpoint file HOT 1
- Have you preprocessed the kinetics dataset? HOT 1
- Where is the code?
- Position IDs in VTN HOT 1
- Reproducing top 1 accuracy in Figure 4 HOT 6
- How to reproduce result in live camera feed without saving as video HOT 2
- Model weights for ViT-B-VTN (1 layer) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from slowfast.