x-plug / mplug-2 Goto Github PK

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video (ICML 2023)

License: Apache License 2.0

Python 99.52% Shell 0.48%

foundation-models mllm multimodal multimodal-pretraining video image-retrieval mplug video-question-answering video-retrieval vqa

mplug-2's Introduction

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video (ICML 2023)

https://arxiv.org/abs/2302.00402

Introduction

we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement. In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement. It is flexible to select different modules for different understanding and generation tasks across all modalities including text, image, and video. mPLUG-2 achieves state-of-the-art or competitive results on a broad range of over 30 downstream tasks, spanning multi-modal tasks of image-text and video-text understanding and generation, and uni-modal tasks of text-only, image-only, and video-only understanding.

News

2023.07.21: Released mPLUG-2 pre-training model and downstream tasks!

Models and Datasets

Pre-trained Models

Model	Visual Backbone	Text Enc Layers	Universal Layers	Fusion Layers	Text Dec Layers	#params	Download
mPLUG-2	ViT-L-14	24	2	6	12	0.9B	mPLUG-2

Pre-train Datasets

	COCO	VG	SBU	CC3M	CC13M	Webvid2M	WikiCorpus
image	113K	100K	860K	3M	10M	2M	20G
text	567K	769K	860K	3M	10M	2M	350G

Downstream Models

VideoQA

Model	Dataset	Accuarcy	Download
mPLUG-2	MSRVTT-QA	48.0	mPLUG-2
mPLUG-2	MSVD-QA	58.1	mPLUG-2

Video Caption

Model	Dataset	CIDER	Download
mPLUG-2	MSRVTT	80.3	mPLUG-2
mPLUG-2	MSVD	165.8	mPLUG-2

Requirements

PyTorch version >= 1.11.0
Install other libraries via

pip install -r requirements.txt

Pre-training

Comming soon.

Fine-tuning

Video Question Answering

Download MSRVTT-QA / MSVD-QA / TGIF datasets from the original websites.
In configs_video/VideoQA_msrvtt_large.yaml, set the paths for the json files and the video paths.
To perform evaluation, run：

sh scripts/inference_videoqa.sh

To perform finetuning, run：

sh scripts/run_videoqa.sh

Video Captioning

Download MSRVTT / MSVD datasets from the original websites.
In configs_video/VideoCaption_msrvtt_large.yaml, set the paths for the json files and the video paths.
To perform evaluation, run：

sh scripts/inference_videocaption.sh

To perform finetuning, run：

sh scripts/run_videocaption.sh

Citation

If you found this work useful, consider giving this repository a star and citing our paper as followed:

@article{Xu2023mPLUG2AM,
  title={mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video},
  author={Haiyang Xu and Qinghao Ye and Ming Yan and Yaya Shi and Jiabo Ye and Yuanhong Xu and Chenliang Li and Bin Bi and Qi Qian and Wei Wang and Guohai Xu and Ji Zhang and Songfang Huang and Fei Huang and Jingren Zhou},
  journal={ArXiv},
  year={2023},
  volume={abs/2302.00402}
}

Acknowledgement

The implementation of mPLUG relies on resources from ALBEF, BLIP, and timm. We thank the original authors for their open-sourcing.

mplug-2's People

Contributors

Stargazers

Watchers

Forkers

vateye xuedipiaofei pooadi chaoannricardo web-logs2 strategist922 billxw ddvd233 wangdeyu ilkerkesen sz1412 valikat jasminejade ahmednoor-19 jeremy-su1 leminhhuan72

mplug-2's Issues

will the pretrain code be released?

Video_id

how do i solve this error??

Code for Video Text Retrieval

Hi, great work!

Can you provide a script to run Video-Text Retrieval with m-PLug2?

Does the inference need to run on 8 A100?

I tried to run inference for video captioning on 1 A100 but got OOM issues. Does the inference need to run on 8 A100 or can it run one one A100? Thanks in advance.

Localizing positions of objects in a scene

Hello
How are you?
Thanks for contributing to this project.
In general, there is NOT ONLY one object in a scene.
So if there are multiple objects in a scene and actions of the objects (ex: person) are different, we need to localize the object's position.
Is it possible to localize positions of all the objects for one video caption?
If it is impossible for right now, do u know any solution or method for this purpose?

Code request

Hi, will you open source the pre-training code for the model and around when will it be available?

Is the zero-shot performance of VideoQA in Table 20 from the model finetuned on ImageQA dataset?

Where can I find the universal layer module?

I checked the file model_video_caption_mplug.py but I find there is no universal layer module in the code, but I see that images are first fed into visual encoders and then fed into text encoders. Does it mean universal layer is actually text encoder?

Finetuned weights for MSRVTT text-video retrieval

Hi, this is really a great work, thank you for sharing the code. Could you please release the MSRVTT finetuned weights for text-video retrieval task? That would be really helpful for me.

how to get train, valid, test

train_file: 'datasets/annotations_all/msrvtt_caption/train.jsonl'
test_file: 'datasets/annotations_all/msrvtt_caption/test.jsonl'
video_root: "datasets/MSRVTT/data/MSRVTT/videos/all"

i get YouTubeClips, AllVideoDescriptions.txt at https://www.cs.utexas.edu/users/ml/clamp/videoDescription/ (Videos, annotation)
and get splits from https://github.com/albanie/collaborative-experts/blob/master/misc/datasets/msvd/README.md

How to perform CIDEr optimization?

Thank you for your excellent work! The paper mentions that CIDEr optimization has been performed for extra 5 epochs in video captioning task. How to run CIDEr optimization using your code?

pretrained_mplug_2.pth

when will the pretrained mplug2 be released?

Is there a way to adapt the model to cartoon video clips?

Hi! Is there a way to adapt the model to cartoon video clips? I do not have cartoon-caption video dataset, however I am searching for a possible way of finetuning the model with cartoon-caption image dataset?

When are you gonna share the weights?

Hello! Thank you for the excellent work. My question is when are you gonna share the weight checkpoints?
Thanks in advance!

Could you provide the JSON file in the video captioning task?

Thank you for your excellent work! However, I noticed that there are missing JSON files in the MSRVTT and MSVD datasets that your code requires. Could you provide them?"

Can't find "language_evaluation"

Hello , when I run the “sh scripts/inference_videocaption.sh” file for the video caption task, the following error was encountered.

But I didn't find "language_evaluation", may I ask where this file can be found?

same, chaos captions

My pred_captions is chaos and always same on one video. I tried to change the model. But still the same caption by two different models?

Evaluation Code?

could you provide evaluation code please? thank you

Double usage of local temporal modeling

Congrats on the awesome work and thanks for sharing it!

I am particularly interested in your temporal modeling. I am investigating your code and it looks like you are using your temporal layer twice in each TemporalBlock. This class has two instances of LocalTemporal: self.lmhra1 and self.lmhra2, one called before and one called after the self-attention layer. Maybe this is connected with an old flag that you have commented out of the code called --double_lmhra?

I might be missing something in the paper, but I don't think that it mentions this double usage of a temporal layer. Could you please confirm whether your numbers come from using one or two temporal layers per block? Could you also please share some insights into the impact of using the temporal layer once vs twice?

Thank you!

mPLUG-2-base pre-training model weights

Can you release the mPLUG-2-base pre-training model weights?

x-plug / mplug-2 Goto Github PK

mplug-2's Introduction

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video (ICML 2023)

Introduction

News

Models and Datasets

Pre-trained Models

Pre-train Datasets

Downstream Models

Requirements

Pre-training

Fine-tuning

Video Question Answering

Video Captioning

Citation

Acknowledgement

mplug-2's People

Contributors

Stargazers

Watchers

Forkers

mplug-2's Issues

Recommend Projects

Recommend Topics

Recommend Org