VTN attention window about slowfast HOT 2 CLOSED

A follow-up question I have is, when the training input has 16 frames, isn't the long former == normal transformer? What's the meaning of setting window size here? No matter the size is 18 or 32, it's not smaller than 16. Then the temporal attention encoder is O(n^2) transformer right? What's the benefit of using Longformer then? Please correct me I misunderstood something.

Thank you.

from slowfast.

Hi @YunYunY,

Thank you for your comments.

To reproduce the results you should use the config and the code in this repository - ATTENTION_WINDOW: [18, 18, 18]. The misalignment in window size between the paper and the code is that different implementations of the Longformer model handle this value slightly different in terms of padding the input sequence and sliding windows size, and we moved from one implementation to the other [1] to [2].

Regarding the second comment, the sliding window is effective when evaluating full videos and the input sequence is much longer than 16 (250 frames for Kinetics-400). In this case, the Longformer attends locally using the sliding window and globally using the CLS token.

from slowfast.