doubleclass / gmm Goto Github PK

View Code? Open in Web Editor NEW

30.0 5.0 2.0 328.62 MB

Generative Multi-modal Models are Good Class Incremental Learners, CVPR 2024 [PyTorch Code]

Python 99.91% Shell 0.09%

gmm's Introduction

Generative Multi-modal Models are Good Class Incremental Learners

This is the official code for our CVPR paper:

Getting Started

Installation

Git clone our repository, creating a python environment and activate it via the following command

git clone https://github.com/DoubleClass/GMM
cd GMM
conda env create -f env_GMM.yaml
conda activate GMM
pip install git+https://github.com/openai/CLIP.git

Vicuna

You can get the LLM Vicuna in huggingface, or you can Dowload it via baidunetdisk. (code: s3pu)

Then set the downloaded vicuna folder path here and the initial checkpoint here

EVA_VIT_G

The code will automatically downloading the eva_vit_g.pth, we alse put it here, you can manually download it and put it in the cache dir: .cache/torch/hub/checkpoints

bert-base-uncased

The code will automatically downloading this, but in case you don't have access to huggingface, we also put it here, you can manually download it and alse put it in cache dir: .cache/huggingface/hub/models--bert-base-uncased

datasets

ImageNet-R

You can download it here

Then set the dataset folder path here

Besides, you need to customize the dataset for the GPT fine-tuning process. We prepare a example here you can follow: download

After downloaded the customized dataset, you can set the data root path here.

Training

After setting all model and dataset config, you can run the following command to start fine-tuning.

python train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml

Testing

After training, you will get a model checkpoint of the last continual learning stage. put the path to scipts in eval_all.sh and specify a results directory.

Then set the results path in the get_score_all.py

Run the script:

bash eval_all.sh

Citation

If you find our repo useful for your research, please consider citing our paper:

@article{cao2024GMM,
  title={Generative Multi-modal Models are Good Class Incremental Learners},
  author={Cao, Xusheng and Lu, Haori and Huang, Linlan and Liu, Xialei and Cheng, Ming-Ming},
  journal={IEEE Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

License

This code is licensed under the Creative Commons Attribution-NonCommercial 4.0 International for non-commercial use only. Please note that any commercial use of this code requires formal permission prior to use.

Contact

For technical questions, please contact [email protected]

gmm's People

Contributors

Stargazers

Watchers

Forkers

sandy4321 xxynov

gmm's Issues

Why Discriminative PT models with memory buffer degrade performance in table 1?

Hi, Thank you for your insightful paper!

I have some questions about Table 1.

You report PT models with a memory buffer, but there is some performance degradation when using exemplars.

However, the official paper reports performance enhancement when adapting exemplars.

I'm curious about this discrepancy. Could you clarify this part?

Thanks.
Regard.

Questions about the loss of training

Hello, that's a great job, can you release your training log or can you say something about the drop in losses you trained for?

Can't test after trainning imagenet

I have a problem with the get_data function in clip_base/dataset.py line 70 giving an error that folder "/imagenet-r/test" does not exist.

Traceback (most recent call last):
File "/home/thhiep/dataspeech/voice_conversion/GMM/batch_eval.py", line 94, in
eval_dataset, classes_names = build_cl_scenarios(
File "/home/thhiep/dataspeech/voice_conversion/GMM/clip_base/datasets.py", line 167, in build_cl_scenarios
scenario = ClassIncremental(
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/continuum/scenarios/class_incremental.py", line 45, in init
self._nb_tasks = self._setup(nb_tasks)
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/continuum/scenarios/class_incremental.py", line 49, in _setup
x, y, _ = self.cl_dataset.get_data()
File "/home/thhiep/dataspeech/voice_conversion/GMM/clip_base/datasets.py", line 71, in get_data
return super().get_data()
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/continuum/datasets/base.py", line 465, in get_data
self.dataset = torchdata.ImageFolder(self.data_path)
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/torchvision/datasets/folder.py", line 309, in init
super().init(
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/torchvision/datasets/folder.py", line 144, in init
classes, class_to_idx = self.find_classes(self.root)
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/torchvision/datasets/folder.py", line 218, in find_classes
return find_classes(directory)
File "/home/thhiep/anaconda3/envs/GMM/lib/python3.9/site-packages/torchvision/datasets/folder.py", line 40, in find_classes
classes = sorted(entry.name for entry in os.scandir(directory) if entry.is_dir())
FileNotFoundError: [Errno 2] No such file or directory: '/home/thhiep/dataspeech/voice_conversion/GMM/imagenet-r/test'

Ask about FSCIL

First I would like to thank you very much for your great work. I'm interested in the FSCIL problem in your work. Can you give me instructions for running Cifar and ImageNet100 5-way 5-shot settings? I want to reproduce the results of table 2 and table 3.
Thank you very much.

Could you release the trained ckpts of the last session?

Thanks for your great work!

Could you release the trained ckpts of the last session on CIFAR-100?

Thanks a lot!