Hi, great work! I'm not seeing any examples of how you convert the input documents to actual prompts for the model. In the paper, the only relevant snippet seems to be:

If models do not provide prompts for task types in MMMU, we conduct prompt engineering on the validation set and use the most effective prompt for the zero-shot setup in the main experiments.

Can you please provide examples of how you formatted the inputs to any of the models you evaluated this on? Thanks!

Note: I see that in #5 you clarify that you follow MMLU, but this seems to contradict the statement in the paper about prompt engineering on the validation set. Can you clarify that in particular?

Thanks a lot for your effort!!!
Can I ask if you are experiencing gpt4v refusing to answer/insisting that there is no image provided when you validate gpt4v with mmmu?
I came across the issue above when I tried to send gpt4v the mmmu dataset...

Thanks a lot again!!!!!!!!!!

Would be interesting to see the actual distribution of answers. Also commands to replicate those numbers?

Hi, I am try to load your dataset with the provided command:
from datasets import load_dataset
dataset = load_dataset("MMMU/MMMU")

However, one error reports as:

ExpectedMoreSplits: {'dev'}

Can u check it?

Thanks for your great work! Do you have any plans to release the evaluation code for LLaVA-v1.5? Looking forward to your reply.

Would it be possible to add MMMU validation to EvalAI?

It'd be great to be able to compare the numbers calculated on the validation set with the ones produced by EvalAI.

Looking through the Correct Examples I came across the genetics example, and it is wrong.

• The numbers doesn't correspond to what's in the image.
• The logic/reasoning shown is regardless incorrect even if numbers had been ok.

It would be interesting to know how you guys are wetting what is considered to be correct?

Hi!
Am I right, that prompts for models were like: "Question <...>, Options: (A) <...>, (B) ..." for multiple-choice and just "Question: <...>" for short-answer? But there're not any defined option letters for multiple-choice in dataset, or you just match each option to the letter in order?

A little confused about this part of the code. The comment means that when an image appears in the options, the image is set to None. But the way images appear in the data options is always as <image_x>, <img='(.*?) '> can't match anything.

Hi I find that the leaderboard has added some new methods, however, for some open source methods, I cannot find either their codes or even their papers. Can u add the relevant link to these methods (e.g., Macro-VL, InfiMM-zephyr) on your leaderboard?

Could I possibly know the evaluation prompt for mPLUG-Owl2? It seems that the subpar performance might be a result of an inappropriate prompt.

For example, the prompt short-answer generation would be:

<|image|>{QUESTION}\nAnswer the question using a single word or phrase.

For multiple-choice question, the prompt would be:

<|image|>{QUESTION}\n{OPTIONS}\nAnswer with the option’s letter from the given choices directly.

Could you please provide the conversion script?

Thank you for your valuable MMMU benchmark.
I have a question regarding your paper. You mentioned in the article that each data point contains at least one image. Then, how were the results for Llama2 7B, FLAN-T5-XXL, Vicuna-13B and GPT-4 Text without OCR or LLaVA Caption obtained?

When I run run_llava.py, I get the following error:
Traceback (most recent call last):
File "/home/MMMU-main/eval/run_llava.py", line 106, in
main()
File "/home/MMMU-main/eval/run_llava.py", line 98, in main
out_samples = run_model(args, samples, model, call_model_engine, tokenizer, processor)
File "/home/MMMU-main/eval/run_llava.py", line 23, in run_model
response = call_model_engine_fn(args, sample, model, tokenizer, processor)
File "/home/MMMU-main/eval/utils/model_utils.py", line 57, in call_llava_engine_df
n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
RuntimeError: The size of tensor a (162) must match the size of tensor b (7) at non-singleton dimension 1
What causes this?

I used this benchmark to measure performance of LLaVA.
The following warning message appeared.

/usr/local/lib/python3.10/dist-packages/PIL/Image.py:981: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images

I think image files did not convert to RGB type image files.

The answer_dict_val.jsonl is very simple and helpful for evaluating the value set of the datasets.

I am wondering if you have any plan to release the answer dict for the test set and dev set as well to simply the evaluation? Thank you!

Great work! Could I ask how are the image types defined and labeled? Thanks!

I was browsing the test set with the Dataset Viewer on HuggingFace (Link) and noticed that, for the Structural Engineering subset of Architecture_and_Engineering, literally every single answer and explanation is equal to "?". Surely this is a bug?

Thanks for your great work! I want to use your data to train a model and I wonder how to transform the parquet dataset format into images and JSON file. Thanks in advance!

Hi, I am try to load your dataset with the provided command:

from datasets import load_dataset
sub_dataset = load_dataset("/data/datasets/MMMU", "Accounting")

However, one error reports as:

Exception has occurred: DataFilesNotFoundError
No (supported) data files found in /MMMU/MMMU
  File "/data/project/bunny/test_eval_mmmu.py", line 4, in <module>
    sub_dataset = load_dataset("/MMMU/MMMU", "Accounting")
datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in /MMMU/MMMU

Can u check it?

Thank you for your great evaluation, we have recently used a training strategy similar to LLava, which co-trains vqa and chat data, resulting in significant improvements. Could you re evaluate our model? https://github.com/THUDM/CogVLM/

Hi, I'm trying to represent LLaVa-1.5-13b on your benchmark using prompts like:

"Question: <image 1> Baxter Company has a relevant range of production between 15,000 and 30,000 units. The following cost data represents average variable costs per unit for 25,000 units of production. If 30,000 units are produced, what are the per unit manufacturing overhead costs incurred?\n Option: (A) $6\n(B) $7\n(C) $8\n(D) $9\nAnswer:"

for multiple-choice (e.g validation_Accounting_1)

Or for open questions:

Question: Using a finite summation, compute the  initial deflection at midspan for the beam in  Figure P8.42. Given: E = 3000 kips/in.2 .  Use 3-ft segments. Assume I = 0.5IG. <image 1>\nAnswer:

(e.g validation_Architecture_and_Engineering_14)
But I'm getting only 32.5% on validation split vs 36.4% (I tried only questions with single input image - 856/900).
What could be the problem?

Originally posted by @teasgen in #5 (comment)

validation_Materials_25

Question:

The density and associated percent crystallinity for two polypropylene materials are as follows: <image 1> Compute the densities of totally crystalline.

Choices:

['0.917 g/cm^3', '0.841 g/cm^3', '0.946 g/cm^3', '0.841 g/cm^3']

Correct answer: B

First of all, answer B and D seem exactly the same, both are 0.841 g/cm^3. Meanwhile, according to the density formula, the final answer for totally crystalline should be around 0.943 (my own calculation), so C seems to be the correct one. Just want to report this.

GPT-4o (“o” for “omni”) is openai's most advanced model. It is multimodal (accepting text or image inputs and outputting text), and it has the same high intelligence as GPT-4 Turbo but is much more efficient—it generates text 2x faster and is 50% cheaper. Additionally, GPT-4o has the best vision and performance across non-English languages of any of openai's models. GPT-4o is available in the OpenAI API to paying customers. https://platform.openai.com/docs/models/gpt-4o

Hi, I am checking your dataset, and the label of "validation_Literature_10" is different from the one in your eval code:

However In the dataset, this data point is:
{'id': 'validation_Literature_10', 'question': 'Refer to the description <image 1>, which term refers to the choices a writer makes, including the combination of distinctive features such as mood, voice, or word choice?', 'options': "['Theme', 'Setting', 'Style', 'Tone']", 'explanation': '', 'image_1': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=340x291 at 0x7FFE9C906620>, 'image_2': None, 'image_3': None, 'image_4': None, 'image_5': None, 'image_6': None, 'image_7': None, 'img_type': "['Icons and Symbols']", 'answer': 'C', 'topic_difficulty': 'Easy', 'question_type': 'multiple-choice', 'subfield': 'Comparative Literature'}
where the GT is "C", however.

Can u check it?

I'm sorry to hear that the homepage is not available right now. I hope you can fix it soon.

Hello, could you please tell me where can I find MMMU_DEV_VAL.tsv for VLMEvalKit. In your huggingface page, I only find .parquet files