ysyecust / osprey Goto Github PK

A part of Along the River During the Qingming Festival (清明上河图)

Spirited Away (千与千寻)

[2023/12/18]🔥 We released the code, osprey-7b model and online demo for Osprey.

Osprey is a mask-text instruction tuning approach that extends MLLMs by incorporating pixel-wise mask regions into language instructions, enabling fine-grained visual understanding. Based on input mask region, Osprey generate the semantic descriptions including short description and detailed description.

Our Osprey can seamlessly integrate with SAM in point-prompt, box-prompt and segmentation everything modes to generate the semantics associated with specific parts or objects.

Click 👇 to try our demo online.

web demo

username: osprey
password: osprey

Point
Box
Everything

1. First install Gradio-Osprey-Demo.
2. Install Segment Anything.

pip install git+https://github.com/facebookresearch/segment-anything.git

3. Download ViT-B SAM model to checkpoints.

4. Run app.py.

cd demo
python app.py --model checkpoint/osprey_7b

1. Clone this repository and navigate to Osprey folder

git clone https://github.com/CircleRadon/Osprey.git
cd Osprey

2. Install packages

conda create -n osprey python=3.10 -y
conda activate osprey
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

1. Convnext-large-CLIP-model🤗: model
2. Osprey-7b model🤗: model

Then change the "mm_vision_tower" in config.json of Osprey-7b model to the path of Convnext-large-CLIP-model.

• Release the checkpoints, inference codes and demo.
• Release the dataset and training scripts.
• Release the evaluation code.
• Release the code for data generation pipeline.

• LLaVA-v1.5: the codebase we built upon.
• SAM: the demo uses the segmentation result from SAM as the input of Osprey.

@misc{Osprey,
  title={Osprey: Pixel Understanding with Visual Instruction Tuning},
  author={Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang and Jianke Zhu},
  year={2023},
  eprint={2312.10032},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}