nightdessert / retrieval_head Goto Github PK

open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality

Home Page: https://arxiv.org/abs/2404.15574

Python 79.31% Jupyter Notebook 20.69%

retrieval_head's Issues

few samples can stably detect retrieval heads?

Hi, thank you for the outstanding work. I observed in your repository the statement "we found that only a few samples can stably detect retrieval heads." Could you please clarify whether the experimental results (Figure 4) presented in your paper were derived from experiments conducted on the following dataset: https://github.com/nightdessert/Retrieval_Head/blob/main/haystack_for_detect/needles.jsonl?

some typing errors

with open('./head_score/llama-2-7b-80k.json') as file:
    head_list = json.loads(file.readline())

head_score_list = [([int(ll) for ll in l[0].split("-")],np.mean(l[1])) for l in head_list.items()]

top_retrieval_heads = [[l[0],  round(np.mean(l[1]), 2)] for l in head_score_list][:10]

Difference between real_needle and needle in needle.jsonl?

Will the choice of needle cause different observational phenomena? What is the difference between these two needles in

Retrieval_Head/haystack_for_detect/needles.jsonl

Line 1 in f59125c

    
           {"needle": "A new report from the WMO shows that records were once again broken, and in some cases smashed, for greenhouse gas levels, surface temperatures, ocean heat and acidification.", "question": "What does a new report from WMO shows ?", "real_needle": "records were once again broken, and in some cases smashed, for greenhouse gas levels, surface temperatures, ocean heat and acidification."}

nightdessert / retrieval_head Goto Github PK

retrieval_head's Issues

few samples can stably detect retrieval heads?

some typing errors

Difference between real_needle and needle in needle.jsonl?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent