Can this idea be used for 3D voxel convolutional NN? about vit-pytorch HOT 9 CLOSED

Thank you a lot

from vit-pytorch.

Thank you so much. My data shape is (31, 20, 20, 20). Voxel size is (20, 20, 20) with 31 channels.

from vit-pytorch.

Sure! It's the same concept, but with an added dimension

from vit-pytorch.

@lucidrains
Thank you for your reply. Will you have any plans to implement it in your library?
By the way, I don't really understand how to deal with channels. For 2D images, they have RGB 3 channels. Thanks a lot

from vit-pytorch.

Not really for this library, as I'd like to keep it image specific. But I'd be happy to share the code snippet you need if you show me the shape of your input tensor. It won't amount to more than 10 lines, before it goes into a standard transformer

from vit-pytorch.

@xuzhang5788 ok, it was a 3 line change https://gist.github.com/lucidrains/213d2be85d67d71147d807737460baf4

from vit-pytorch.

@lucidrains Thank you very much. Does Linformer library need some changes for 3D? If I want to have 10 patches, is it okay to change
efficient_transformer = Linformer(
dim=128,
seq_len=49+1, # 7x7 patches + 1 cls-token
depth=12,
heads=8,
k=64
)

into

efficient_transformer = Linformer(
dim=128,
seq_len=1000+1, # 7x7 patches + 1 cls-token
depth=12,
heads=8,
k=64
)

from vit-pytorch.

@xuzhang5788 you just have to make sure the sequence length is correct

yup, 10 patches would be 10 ** 3 + 1 (for cls token)

from vit-pytorch.

for linformer, k is recommended to be around 256 at that length

from vit-pytorch.