This is the source code for the paper: Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts (ICML 2024).

• Python 3.8
• PyTorch 2.0
• Torchvision 0.15
• Tensorboard

• Other dependencies are listed in requirements.txt.

To install requirements, run:

conda create -n lift python=3.8 -y
conda activate lift
conda install pytorch==2.0.0 torchvision==0.15.0 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install tensorboard
pip install -r requirements.txt

We encourage installing the latest dependencies. If there are any incompatibilities, please install the dependencies with the following versions.

numpy==1.24.3
scipy==1.10.1
scikit-learn==1.2.1
yacs==0.1.8
tqdm==4.64.1
ftfy==6.1.1
regex==2022.7.9
timm==0.6.12

Most experiments can be reproduced using a single GPU with 20GB of memory (larger models such as ViT-L require more memory).

• To further reduce the GPU memory cost, gradient accumulation is recommended. Please refer to Usage for detailed instructions.

# run LIFT on CIFAR-100-LT (with imbalanced ratio=100)
python main.py -d cifar100_ir100 -m clip_vit_b16 adaptformer True

By running the above command, you can automatically download the CIFAR-100 dataset and run the method (LIFT).

Download the dataset Places, ImageNet, and iNaturalist 2018.

Put files in the following locations and change the path in the data configure files in configs/data:

• Places

Path/To/Dataset
├─ train
│  ├─ airfield
|  |  ├─ 00000001.jpg
|  |  └─ ......
│  └─ ......
└─ val
   ├─ airfield
   |  ├─ Places365_val_00000435.jpg
   |  └─ ......
   └─ ......

• ImageNet

Path/To/Dataset
├─ train
│  ├─ n01440764
|  |  ├─ n01440764_18.JPEG
|  |  └─ ......
│  └─ ......
└─ val
   ├─ n01440764
   |  ├─ ILSVRC2012_val_00000293.JPEG
   |  └─ ......
   └─ ......

• iNaturalist 2018

Path/To/Dataset
└─ train_val2018
   ├─ Actinopterygii
   |  ├─ 2229
   |  |  ├─ 2c5596da5091695e44b5604c2a53c477.jpg
   |  |  └─ ......
   |  └─ ......
   └─ ......

To reproduce the main result in the paper, please run

# run LIFT on ImageNet-LT
python main.py -d imagenet_lt -m clip_vit_b16 adaptformer True

# run LIFT on Places-LT
python main.py -d places_lt -m clip_vit_b16 adaptformer True

# run LIFT on iNaturalist 2018
python main.py -d inat2018 -m clip_vit_b16_peft adaptformer True num_epochs 20

For other experiments, please refer to scripts for reproduction commands.

To train and test the proposed method on more settings, run

python main.py -d [data] -m [model] [options]

The [data] can be the name of a .yaml file in configs/data, including imagenet_lt, places_lt, inat2018, cifar100_ir100, cifar100_ir50, cifar100_ir10, etc.

The [model] can be the name of a .yaml file in configs/model, including clip_rn50, clip_vit_b16, in21k_vit_b16, etc.

Note that using only -d and -m options denotes only fine-tuning the classifier. Please use additional [options] for more settings.

• To apply lightweight fine-tuning methods, add options like lora True, adaptformer True, etc.

• To apply test-time ensembling, add tte True.

Moreover, [options] can facilitate modifying the configure options in utils/config.py. Following are some examples.

• To specify the root path of datasets, add root Path/To/Datasets.

• To change the output directory, add an option like output_dir NewExpDir. Then the results will be saved in output/NewExpDir.

• To assign a single GPU (for example, GPU 0), add an option like gpu 0.

• To apply gradient accumulation, add micro_batch_size XX. This can further reduce GPU memory costs. Note that XX should be a divisor of batch_size.

• To test an existing model, add test_only True. This option will test the model trained by your configure file. To test another model, add an additional option like model_dir output/AnotherExpDir.

• To test an existing model on the training set, add test_train True.

You can also refer to scripts for example commands.

We thank the authors for the following repositories for code reference: [OLTR], [Classifier-Balancing], [Dassl], [CoOp].

If you find this repo useful for your work, please cite as:

@inproceedings{shi2024longtail,
  title={Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts},
  author={Jiang-Xin Shi and Tong Wei and Zhi Zhou and Jie-Jing Shao and Xin-Yan Han and Yu-Feng Li},
  booktitle={Proceedings of the 41st International Conference on Machine Learning},
  year={2024}
}