openbmb / minicpm-v Goto Github PK
View Code? Open in Web Editor NEWMiniCPM-V 2.6: A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone
License: Apache License 2.0
MiniCPM-V 2.6: A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone
License: Apache License 2.0
RT
看到 swift 里面有 V1 的微调代码,请问在 V2.0 上可以直接使用么,还是需要重新开发一下?
重装flash_attn_2也不能解决。
还是同样错误
Thanks for the great work!
I am wondering where we can find the training data composition for MiniCPM-v2? Specifically, if any distillation from other LMMs is used?
如题,感谢🙏
我看示例截图里都是英语的.
我观察到minicpm的llm有多个版本,例如fp32,bf16,dpo和sft。
请问多模态模型是使用哪一个llm呢?
Such as data/training strategy/ablation study etc?
Hi, and thanks for this fantastic work!
I am using your models for a scientific project, which requires details about the training procedure and training data. I haven't found a paper or a technical report specifically for this work. Did I miss something?
Is the training procedure and data the same as in https://arxiv.org/pdf/2308.12038.pdf ?
Can you please share some details or a technical report?
Thanks :-)
the demo looks like only support one image in one prompt. i wonder does it support multi-image interleaved conversation? e.g. input 2 images at once and compare which is brighter
Hi, Is it possible convert omniLLM-3B (miniCPM-V) to Quantization version ( e,g: GPTQ ) ?
Or, just opensource the training data
请问一下演示的demo部分的代码大概什么时候开源呢?可以先说一下框架和实现的思路吗?
遍历图片时总是复制前面的输出,是因为跑多个图片预测时会缓存前面的输出结果?
而单独对每个图片进行推断,中文ocr效果又很差。是不支持中文ocr识别吗
Hi, I got out of memory/memory leak after after some calls. Here the script
model = AutoModel.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True, cache_dir="./cache", torch_dtype=torch.bfloat16)
model = model.to(device='cuda', dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True, cache_dir="./cache")
files = ['a.jpg', 'b.jpg', 'c.jpg', ...]
for image_path in files:
msgs = [{'role': 'user', 'content': prompt}]
image = Image.open(image_path).convert('RGB')
res, context, _ = model.chat(
image=image,
msgs=msgs,
context=None,
tokenizer=tokenizer,
sampling=False,
temperature=0.0
)
print(res)
使用Mac部署的时候 提示没有使用CUDA的错误
`
(OmniLMM) crz@crzdeMacBook-Air OmniLMM % pip install flash_attn
Collecting flash_attn
Downloading flash_attn-2.5.2.tar.gz (2.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 882.4 kB/s eta 0:00:00
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
fatal: not a git repository (or any of the parent directories): .git
/private/var/folders/fn/bz39wtv175s83wx3cm11lptm0000gn/T/pip-install-k91r8ka8/flash-attn_fccd9cf359024ff8acc3369bdc82e45f/setup.py:78: UserWarning: flash_attn was requested, but nvcc was not found. Are you sure your environment has nvcc available? If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc.
warnings.warn(
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/private/var/folders/fn/bz39wtv175s83wx3cm11lptm0000gn/T/pip-install-k91r8ka8/flash-attn_fccd9cf359024ff8acc3369bdc82e45f/setup.py", line 133, in
CUDAExtension(
File "/Users/crz/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1048, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/Users/crz/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1179, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/Users/crz/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2223, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
torch.__version__ = 2.0.1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.`
请问MiniCPM-V如何使用流式的方式生成?像是使用transformers的streamer之类的。
We got different benchmark results while evaluation the MiniCPM-V 2.0 model provided in huggingface. For example, we got only 66 on TextVQA, which is different from 74.1 provided in the technique report. We directly use the chat
api in modeling_minicpmv.py like this.
output_ids, context, _ = model.chat(
image=image_tensor,
msgs=msgs,
context=None,
tokenizer=tokenizer,
sampling=True,
temperature=0.7
)
Would you please provide the detailed hyperparameters for the evaluation?
会支持VLLM推理吗
Apologies for asking this in the issues tab.
Is this model capable of chatting without image input? I see in the benchmarks you have some models that can take both image+text and just text. I'm looking to use this model for a project, but I'm unsure if it'll work for my needs.
方便提供openai_api接口吗,这样我就可以做成服务调用了,VLLM就更好了,听说这周就可以实现吗?
我下载了3b模型跑了官方的MME测试集,使用prompt为官方的默认prompt,算出来的分数为1378.67,和给出的1452差别较大。请问是测试结果使用的prompt不一致导致的吗?
请问支持中文数据微调/推理吗
Hello. I'm getting followed error when i try to run the demo of MiniCPM-V on Ubuntu 22.04:
(OmniLMM) markusc@ubuntu2:~/github/OmniLMM$ /home/markusc/anaconda3/envs/OmniLMM/bin/python /home/markusc/github/OmniLMM/infer.py Loading checkpoint shards: 100%|████████████████████████████████████████████| 2/2 [00:01<00:00, 1.94it/s] Traceback (most recent call last): File "/home/markusc/github/OmniLMM/infer.py", line 10, in <module> answer = chat_model.process(inputs) AttributeError: 'OmniLMMChat' object has no attribute 'process'
Apart from importing the json module and changing the model path to openbmb/MiniCPM-V, I did not make any modifications to the original code. As followed:
`from chat import OmniLMMChat, img2base64
import json
chat_model = OmniLMMChat('openbmb/MiniCPM-V')
im_64 = img2base64('./assets/worldmap_ck.jpg')
msgs = [{"role": "user", "content": "What is interesting about this image?"}]
inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.process(inputs)
print(answer)
msgs.append({"role": "assistant", "content": answer})
msgs.append({"role": "user", "content": "Where is China in the image"})
inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.process(inputs)
print(answer)`
Thanks in advance.
数学推理出错
prompt: 1+1/sqrt(2)=?
completion: 1 + 1/sqrt(2) 可以转换为一个数学表达式。
首先,我们可以将根号下的 2 化简为 √2。
然后,我们可以将 1 视为一个常数,并将其加到 1/√2。为了将 1/√2 转换为一个分数,我们可以将其看作是一个带根号的分数。我们可以将根号下的 2 平方,得到 2。因此,1/√2 可以表示为 1/(√2 * √2),即 1/(2 * 2)。
现在,我们可以将 1 + 1/√2 表示为:
1 + 1/√2 = 1 + 1/2
接下来,我们可以将这个表达式转换为一个带分数:
1 + 1/√2 = 2/2 + 1/2 = 3/2
所以,1 + 1/√2 的结果是 3/2 或 1 1/2。
您好,感谢发布这么优秀的模型。请问什么时候可以提供微调代码呢?
I installed conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c
pytorch -c nvidia
but an error message showed up, RuntimeError: cutlassF: no kernel found to launch! , it indicates that my cuda is not the correct verision , I downloaded the model from modelscope
我根据requirements.txt安装了依赖 但是在运行代码时 系统提示我需要安装cuda , 但我安装了cuda后依然报错,是说我的cuda版本与你们的不一致,我是自己下载了模型,并且在使用
训练过程细节有吗?
我注意到没有finetune的代码,是不是可以使用viscpm的代码? 还是说team后续会放出?
I have tried the multi card inference but failed, it seems the layers are not dispatched correctly.
with init_empty_weights():
model = OmniLMMForCausalLM.from_pretrained(model_name, tune_clip=True, torch_dtype=torch.bfloat16)
model = load_checkpoint_and_dispatch(model, model_name, dtype=torch.bfloat16,
device_map="balanced", no_split_module_classes=['Eva','MistralDecoderLayer', 'ModuleList', 'Resampler']
)
So it would be convenience to provide a quantization model for low memory cards, it may be nice if under 20GB single card or 40GB multi cards.
关于MiniCPM-V,我注意到Github上报告MMMU val 34.7,而官方leaderboard是37.2?
Hello, is it possible (will it be) to fine tune with peft lora and quantization?
请问,支持detection返回bbox吗?很多多模态模型已经支持了,且性能也很优秀,为什么cpmv我测了感觉不支持。
但是cpmv对空间位置关系的感知能力也很强,是什么原因导致放弃了训练这一功能呢?
如果加上detect的预训练,做finetune时有bbox输入对应文本,是否能让模型更容易获取新知识?
MLCChat failed
Stack trace:
org.apache.tvm.Base$TVMError: TVMError: OpenCL build error for device=0x78b138eaa8
Error: CL_OUT_OF_HOST_MEMORY
Pass
Stack trace:
File "/Users/acha/Desktop/mlc-MiniCPM/3rdparty/tvm/src/runtime/library_module.cc", line 78
at org.apache.tvm.Base.checkCall(Base.java:173)
at org.apache.tvm.Function.invoke(Function.java:130)
at com.modelbest.mlcllm.ChatModule.image(ChatModule.java:64)
at com.modelbest.minicpm.AppViewModel$ChatState$requestImage$f$1$1.invoke(AppViewModel.kt:637)
at com.modelbest.minicpm.AppViewModel$ChatState$requestImage$f$1$1.invoke(AppViewModel.kt:637)
at com.modelbest.minicpm.AppViewModel$ChatState.callBackend(AppViewModel.kt:516)
at com.modelbest.minicpm.AppViewModel$ChatState.requestImage$lambda$4(AppViewModel.kt:637)
at com.modelbest.minicpm.AppViewModel$ChatState.$r8$lambda$-81FfOHgL43Kw6mcY6rLiYQKEuE(Unknown Source:0)
at com.modelbest.minicpm.AppViewModel$ChatState$$ExternalSyntheticLambda2.run(Unknown Source:12)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at java.lang.Thread.run(Thread.java:1012)
Error message:
TVMError: OpenCL build error for device=0x78b138eaa8
Error: CL_OUT_OF_HOST_MEMORY
Pass
Stack trace:
File "/Users/acha/Desktop/mlc-MiniCPM/3rdparty/tvm/src/runtime/library_module.cc", line 78
(OmniLMM) [zhuguibo@GPU1 OmniLMM]$ python example.py
Traceback (most recent call last):
File "/mnt/data/dataProcess/LLM_Models/LLM_code/minicpm/OmniLMM/example.py", line 52, in
option1()
File "/mnt/data/dataProcess/LLM_Models/LLM_code/minicpm/OmniLMM/example.py", line 5, in option1
chat_model = OmniLMMChat('/mnt/data/dataProcess/LLM_Models/MiniCPM-V-2.0')
File "/mnt/data/dataProcess/LLM_Models/LLM_code/minicpm/OmniLMM/chat.py", line 169, in init
self.model = OmniLMM3B(model_path)
File "/mnt/data/dataProcess/LLM_Models/LLM_code/minicpm/OmniLMM/chat.py", line 141, in init
self.model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(dtype=torch.bfloat16)
File "/mnt/data/app/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/mnt/data/app/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1091, in from_pretrained
config_class = get_class_from_dynamic_module(
File "/mnt/data/app/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 500, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module.replace(".py", ""))
File "/mnt/data/app/anaconda3/envs/OmniLMM/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 200, in get_class_in_module
module = importlib.import_module(module_path)
File "/mnt/data/app/anaconda3/envs/OmniLMM/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers_modules.MiniCPM-V-2'
MiniCPM-V和OmniLMM都支持中文吗?
http://120.92.209.146/ 这个demo地址的webui我想测试下。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.