Could you please provide the code for generating samples of Math.Find, Math.Calc, Code.RUN and Code.debug?
I want to generate some test sample with shorter length, since my model only support 32k context length.
Thanks a lot!

Hi,

Yi-200K has show-cased retriving knowledge from ThreeBodyProblem 2, but its's long context ability has not been tested like Claude2 and GPT4-128K.

Would Yi also be evaluated? It is not mentioned in your readme though

Thanks

math_find的label是个列表，但是要求是int或者float类型

Awesome work! Would you have any plans to release the data source? I'm very curious about where the super-long (>100k) code and math data from.

Thanks for great work!
I find that in-context examples in mathcalc templates are not consistent with the huggingface dataset preview(Whether the first value is the first expected output in the answer). I am wondering whether there is a mistake in templates, which results in models' low performance on mathcalc dataset. Besides, the reason why GPT4 always output the first number as the first value may also be attributed to this inconsistency.

Hey great efforts on constructing InfiniBench! One quick question I've noticed:

In a recent upload to HF dataset https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench/commit/f2fd8f04ea3af8304b88de2c58bd33887bcccdb8

You changed 20 questions and answers compared to the original commit in: https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench/commit/bae90aaac350652b7c7292dcf556bdd52542929b

Consequently some numbers based on the previous / current dataset may not be directly comparable. May I ask what are the considerations behind this?

Thanks!

有几个问题，还麻烦您解答，非常感谢：

1. 该bench支持64K甚至32K的长文本测评吗？
2. 如果支持的话，是需要修改MAX_POSITION_ID 和TRUNCATE_LEN 中的128为对应模型的支持长度吗，如64k模型修改为TRUNCATE_LEN =64*1024？
3. 测试样本中根据长度做了切块，代码逻辑与LongBench是一致的，请问我理解的对吗？

期待您的回复，不胜感激!

执行python3 eval_yarn_mistral.py --task passkey --model_path ../Yarn-Mistral-7b-128k/ 出错

出错信息如下：
src/eval_utils.py", line 204, in create_prompt
"prompt": eg["prompt"],
KeyError: 'prompt'

如果注释掉这行，mistral的生成结果非常奇怪：
==== Evaluation ====

examples: 590

Start index: 0
Stop index: 590
Verbose: False
Max tokens: 6
====== Example 0 ======
Truncating...

tokens before: 125353

tokens after: 125353

seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id to eos_token_id:2 for open-end generation.
Chunked generation: green.

What is
====== Example 1 ======
Truncating...

tokens before: 125353

tokens after: 125353

seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id to eos_token_id:2 for open-end generation.
Chunked generation: green.

What is
====== Example 2 ======
Truncating...

tokens before: 125353

tokens after: 125353

seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id to eos_token_id:2 for open-end generation.
Chunked generation: 89415
====== Example 3 ======
Truncating...

tokens before: 125353

tokens after: 125353

seq_len: 125353
Number of chunks: 980, generating...
Setting pad_token_id to eos_token_id:2 for open-end generation.
Chunked generation: green.

What is

How is GPT4 run if the API has a hard-cutoff of 128k?
The EN.QA and EN.MC dataset itself looks to be more than 128k tokens by itself.

Am I missing something?

Very useful benchmark! May I ask how long did it take when you had inference on these tasks using YaRN-Mistral-7B? Did you only use one A100 80GB GPU for inference?

If so, how do you handle more than 32K tokens?

你好！我想问一下，除了retrieval这种可以通过程序来自动构建的数据集，其他部分全是人工标注的吗？有没有模型辅助生成、人工check的部分？

especially for the Multi Needle in a Haystack

I have seen https://github.com/OpenBMB/InfiniteBench/blob/main/scripts/eval_rwkv.sh

python eval_rwkv.py --task ${task}

btw there is no eval_rwkv.py.
how to evaluate RWKV or Jamba with your scripts?
thanks.

https://github.com/OpenBMB/InfiniteBench/blob/main/src/compute_scores.py#L238

1. only one reference label is used for comparison, better loop around each answer in label, e.g., label=['ECKER', 'COMMANDER BILL ECKER'];
2. prediction phrase is splitted into words for comparison, which means:
   pred=STAMP PAID, label=['STAMP PAID'], score=0.0, data_name=longdialogue_qa_eng

and

pred=ADRIAN, label=['ADRIAN'], score=1.0, data_name=longdialogue_qa_eng

the label of Math.Calc in the dataset is like this:

[[79,
  59,
  73,
  78,
  43,
  21,
  ...,
  46,
  59,
  ...]]

It is List[List]
but the code in compute_scores.py:

def get_score_one_math_calc(pred, label, model_name: str) -> float:
    assert isinstance(label, list), f"Expected list, got {type(label)}"
    # assert isinstance(pred, list), f"Expected list, got {type(pred)}"
    pred_nums = []
    pred_list = re.split("[^0-9]", pred)
    for item in pred_list:
        if item != "":
            pred_nums.append(int(item))

    # Our prompts makes GPT4 always output the first number as the first value
    # in the predicted answer.
    if model_name == "gpt4":
        pred_nums = pred_nums[1:]

    cnt = 0
    for i in range(len(label)):
        if i >= len(pred_nums):
            break
        if label[i] == pred_nums[i]:
            cnt += 1
        else:
            break
    return cnt / len(label)

The code just compare the first int of the pred_nums with the first item of the label which is a list.

The label is got in get_answer.

In the code of get_answer, it does not any special process of the task 'math_calc', why your result the answer is a list, but my result the groudtruth is a List[List]?

def get_answer(eg: dict, data_name: str):
    if data_name in ["code_debug", "longbook_choice_eng"]:
        OPTIONS = "ABCD"
        if isinstance(eg["answer"], str):
            ret = [eg["answer"], OPTIONS[eg['options'].index(eg["answer"])]]
        elif isinstance(eg["answer"], list):
            if len(eg["answer"]) == 1:
                ret = [eg["answer"][0], OPTIONS[eg['options'].index(eg["answer"][0])]]
            elif len(eg["answer"]) == 2 and eg["answer"][1] in ['A', 'B', 'C', 'D']:
                ret = eg['answer']
            else:
                raise ValueError
        else:
            raise ValueError
        return ret

    return eg["answer"]