关于提升数据集测试有效性的建议 about longbench HOT 4 OPEN

哦哦 我明白你的意思了 你的意思是 人类语言天然的就存在evidence偏向于在首尾？ 好吧，这样就难以区分哪些是真正有效的策略了。

from longbench.

谢谢你的建议。longbench中synthetic tasks是这样随机构造的，即我们把evidence的段落放在context的随机位置。在其他任务中，为了保证和真实场景分布一致，我们避免用这种人造方式改变原先的context。这种答案分布的bias在真实场景中也往往是存在的——例如文章的开头、末尾一般更加重要。

from longbench.

感谢回复，但是我试着只使用最末尾的1k token进行推理，精度和使用全量数据接近，这不合理啊。

from longbench.

In our practical testing process, we’ve encountered a similar issue.
We’ve noticed that various methods claiming to compress the sequence dimensions of KV caches (such as different heads drop different part of the sequence) and approaches for handling long sequence inference (like streamLLM) perform well on LongBench. This is because they achieve high scores by retaining only a small portion of the text at the end of the sequence. However, tasks like ‘finding a needle in a haystack’ can more accurately evaluate a model’s capabilities because they hide the ‘needle’ in different positions. Unfortunately, ‘finding a needle in a haystack’ doesn’t cover all aspects comprehensively. If possible, I hope the authors can enhance the LongBench mechanism.

from longbench.