Temporal attention questions about devis HOT 2 CLOSED

Hello, thanks a lot for your interest!

Yes, as you mention, the encoder should work the same as the decoder when INSTANCE_AWARE_ATTENTION=False. The current difference is that on the encoder we assume that all the resolutions will be valid (valid_ratios all ones), so we pick the coordinates of the first one. Note that this assumption should hold true as batch size is fixed to 1.

Note that for each pixel position, the reference_point will be the same for each resolution and for each frame, thus on the decoder the result of reference_points[t].repeat(1, self.t_window, 1)[None, :,None, :, None] has the exact same location on the 4th axis that holds num_res_lvls *num_temporal_frames positions (4x5 = 20 on the default implementation).

In the current implementation, the encoder has just a single number instead of being repeated 20 times, so the result when summed with the corresponding offset should not make any difference between both encoder and decoder if I am not wrong.

To sum up, it would be more correct as you say to have used the same operation as on the decoder for consistency, but should not make any difference in terms of results.

Hope I have helped, if you have any more doubts feel free to ask me! Thanks.

from devis.

Thanks a lot!

from devis.