Official PyTorch implementation of CLE-ViT: Contrastive Learning Encoded Transformer for Ultra-Fine-Grained Visual Categorization (IJCAI 2023).
If you use the code in this repo for your work, please cite the following bib entries:
Ultra-fine-grained visual classification (ultra-FGVC) targets at classifying sub-grained categories of fine-grained objects. This inevitably requires discriminative representation learning within a limited training set. Exploring intrinsic features from the object itself, e.g. , predicting the rotation of a given image, has demonstrated great progress towards learning discriminative representation. Yet none of these works consider explicit supervision for learning mutual information at instance level. To this end, this paper introduces CLE-ViT, a novel contrastive learning encoded transformer, to address the fundamental problem in ultra-FGVC. The core design is a self-supervised module that performs self-shuffling and masking and then distinguishes these altered images from other images. This drives the model to learn an optimized feature space
that has a large inter-class distance while remaining tolerant to intra-class variations. By incorporating this self-supervised module, the network acquires more knowledge from the intrinsic structure of the input data, which improves the generalization ability without requiring extra manual annotations. CLE-ViT demonstrates strong performance on 7 publicly available datasets, demonstrating its effectiveness in the ultra-FGVC task.
Please use the command below to create the environment for CLE-ViT.
$ conda env create -f env.yaml
- Get models in this link: Swin-B, Swin-S...
wget https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pthYou can download the datasets from the links below:
Using the scripts on scripts directory to train the model, e.g., train on SoybeanGene dataset.
$ sh scripts/run_gene.sh
Password: r5zr
Our project references the codes in the following repos. Thanks for thier works and sharing.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent