chengyuegongr / patchvisiontransformer Goto Github PK

Hey, this is an interesting work.
Just wondering why using the absolute cosine similarity instead of the original one.
Thanks!

Hi, I train a vit_base_patch32 model with reslution 224 on imagenet, the valid acc comes up to 73.38%, then I dump all transformer blocks' outputs, calculate cosine similarity between them as mentioned in paper, I cannot have the same result in paper, here is my result. the 0,1,2,3,4,5,6,7,8,9,10,11 is layer depth, right is the value of cosine similarity
0 ---> 0.5428178586547845
1 ---> 0.6069238793659271
2 ---> 0.30199843167793006
3 ---> 0.26388993740273486
4 ---> 0.26132955026320265
5 ---> 0.24258930458215844
6 ---> 0.20970458839482967
7 ---> 0.21119057468517677
8 ---> 0.22155304307901189
9 ---> 0.23545575648548187
10 ---> 0.2329663067175004
11 ---> 0.22496230679589768

Hi, I find aux_class is setted as None in your code, so have you use patch_loss in your work?

target_lst, mask_lst, targets = aux_class

what these variables mean: target_lst, mask_lst, targets?

Thank you very much!

Hi, thanks for sharing code!
I'd like to try your code with mmsegmentation.
But I can't find which part is the different with original swin.
Shortly, I don't know where the diverse part is

Sincerely

Hello, I want to know what the 'low_order' and 'high_order' represent in the 'similarity' function in models.py and how to set the 'high_k'?

I also wonder which parts of the codes present the L_cos in the paper?

Thank you very much!

/PatchVisionTransformer/deit/models.py", line 232, in similarity
high_order = (patches.mean(dim=(2, 3), deepdim=True) - patches / (img_size ** 2)) * (img_size ** 2) / (img_size ** 2 - 1)
TypeError: mean() received an invalid combination of arguments - got (deepdim=bool, dim=tuple, ), but expected one of:

• (*, torch.dtype dtype)
  didn't match because some of the keywords were incorrect: deepdim, dim
• (tuple of names dim, bool keepdim, *, torch.dtype dtype)
• (tuple of ints dim, bool keepdim, *, torch.dtype dtype)

According to your instruction in readme, do you need to train 600 epoch totally to get the results shown in the paper?