openbmb / infinitebench Goto Github PK
View Code? Open in Web Editor NEWCodes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
License: MIT License
Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
License: MIT License
执行python3 eval_yarn_mistral.py --task passkey --model_path ../Yarn-Mistral-7b-128k/ 出错
出错信息如下:
src/eval_utils.py", line 204, in create_prompt
"prompt": eg["prompt"],
KeyError: 'prompt'
如果注释掉这行,mistral的生成结果非常奇怪:
==== Evaluation ====
Start index: 0
Stop index: 590
Verbose: False
Max tokens: 6
====== Example 0 ======
Truncating...
seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id
to eos_token_id
:2 for open-end generation.
Chunked generation: green.
What is
====== Example 1 ======
Truncating...
seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id
to eos_token_id
:2 for open-end generation.
Chunked generation: green.
What is
====== Example 2 ======
Truncating...
seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id
to eos_token_id
:2 for open-end generation.
Chunked generation: 89415
====== Example 3 ======
Truncating...
seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id
to eos_token_id
:2 for open-end generation.
Chunked generation: green.
What is
Very useful benchmark! May I ask how long did it take when you had inference on these tasks using YaRN-Mistral-7B? Did you only use one A100 80GB GPU for inference?
你好!我想问一下,除了retrieval这种可以通过程序来自动构建的数据集,其他部分全是人工标注的吗?有没有模型辅助生成、人工check的部分?
If so, how do you handle more than 32K tokens?
Hi,
Yi-200K has show-cased retriving knowledge from ThreeBodyProblem 2, but its's long context ability has not been tested like Claude2 and GPT4-128K.
Would Yi also be evaluated? It is not mentioned in your readme though
Thanks
有几个问题,还麻烦您解答,非常感谢:
MAX_POSITION_ID
和TRUNCATE_LEN
中的128为对应模型的支持长度吗,如64k模型修改为TRUNCATE_LEN =64*1024
?期待您的回复,不胜感激!
When I try to run the following code in colab:
from datasets import load_dataset
dataset = load_dataset("xinrongzhang2022/InfiniteBench")
I get the following error:
DatasetGenerationCastError: An error occurred while generating the dataset
All the data files must have the same columns, but at some point there are 1 missing columns ({'options'})
This happened while the json dataset builder was generating data using
hf://datasets/xinrongzhang2022/InfiniteBench/kv_retrieval.jsonl (at revision 2c3c9fe62808833ab783026bbf8e7a47539a28c6)
Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)
especially for the Multi Needle in a Haystack
I have seen https://github.com/OpenBMB/InfiniteBench/blob/main/scripts/eval_rwkv.sh
python eval_rwkv.py --task ${task}
btw there is no eval_rwkv.py.
how to evaluate RWKV or Jamba with your scripts?
thanks.
Could you please provide the code for generating samples of Math.Find, Math.Calc, Code.RUN and Code.debug?
I want to generate some test sample with shorter length, since my model only support 32k context length.
Thanks a lot!
Hey great efforts on constructing InfiniBench! One quick question I've noticed:
In a recent upload to HF dataset https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench/commit/f2fd8f04ea3af8304b88de2c58bd33887bcccdb8
You changed 20 questions and answers compared to the original commit in: https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench/commit/bae90aaac350652b7c7292dcf556bdd52542929b
Consequently some numbers based on the previous / current dataset may not be directly comparable. May I ask what are the considerations behind this?
Thanks!
Awesome work! Would you have any plans to release the data source? I'm very curious about where the super-long (>100k) code and math data from.
the label of Math.Calc in the dataset is like this:
[[79,
59,
73,
78,
43,
21,
...,
46,
59,
...]]
It is List[List]
but the code in compute_scores.py:
def get_score_one_math_calc(pred, label, model_name: str) -> float:
assert isinstance(label, list), f"Expected list, got {type(label)}"
# assert isinstance(pred, list), f"Expected list, got {type(pred)}"
pred_nums = []
pred_list = re.split("[^0-9]", pred)
for item in pred_list:
if item != "":
pred_nums.append(int(item))
# Our prompts makes GPT4 always output the first number as the first value
# in the predicted answer.
if model_name == "gpt4":
pred_nums = pred_nums[1:]
cnt = 0
for i in range(len(label)):
if i >= len(pred_nums):
break
if label[i] == pred_nums[i]:
cnt += 1
else:
break
return cnt / len(label)
The code just compare the first int of the pred_nums with the first item of the label which is a list.
The label is got in get_answer
.
In the code of get_answer, it does not any special process of the task 'math_calc', why your result the answer is a list, but my result the groudtruth is a List[List]?
def get_answer(eg: dict, data_name: str):
if data_name in ["code_debug", "longbook_choice_eng"]:
OPTIONS = "ABCD"
if isinstance(eg["answer"], str):
ret = [eg["answer"], OPTIONS[eg['options'].index(eg["answer"])]]
elif isinstance(eg["answer"], list):
if len(eg["answer"]) == 1:
ret = [eg["answer"][0], OPTIONS[eg['options'].index(eg["answer"][0])]]
elif len(eg["answer"]) == 2 and eg["answer"][1] in ['A', 'B', 'C', 'D']:
ret = eg['answer']
else:
raise ValueError
else:
raise ValueError
return ret
return eg["answer"]
Thanks for great work!
I find that in-context examples in mathcalc templates are not consistent with the huggingface dataset preview(Whether the first value is the first expected output in the answer). I am wondering whether there is a mistake in templates, which results in models' low performance on mathcalc dataset. Besides, the reason why GPT4 always output the first number as the first value may also be attributed to this inconsistency.
Are the GPT4 results evaluated on a different set of longbook_qa_eng
? The 'ground_truth' fields in results/gpt4/preds_longbook_qa_eng.jsonl don't seem match with ground_truth in results/chatglm3/preds_longbook_qa_eng.jsonl
How is GPT4 run if the API has a hard-cutoff of 128k?
The EN.QA and EN.MC dataset itself looks to be more than 128k tokens by itself.
Am I missing something?
https://github.com/OpenBMB/InfiniteBench/blob/main/src/compute_scores.py#L238
pred=STAMP PAID, label=['STAMP PAID'], score=0.0, data_name=longdialogue_qa_eng
and
pred=ADRIAN, label=['ADRIAN'], score=1.0, data_name=longdialogue_qa_eng
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.