KeyError: 'Subject_Aliasing_acc' about easyedit HOT 9 CLOSED

1266th,in 'pre' dict,without 'Subject_Aliasing_acc'

1265th and above,in 'pre' dict,with 'Subject_Aliasing_acc'

from easyedit.

Do you mean that the data for item 1265 doesn't have Subject_Aliasing_acc in the 'pre' , but does have Subject_Aliasing_acc in the 'post' ?

from easyedit.

in Llama-2-7b-ms_recent_pre_edit.json
pre 1266,

Subject_Aliasing_acconly 629

from easyedit.

this is relevant issue [https://github.com//issues/274#issue-2324392601]

from easyedit.

Hello, Subject_Aliasing_acc is a metric under portability, and not all edited data have this. However, locality evaluates the model's locality, and every data point will have it. Our evaluation script has already handled cases where Subject_Aliasing_acc is missing. Could you please provide your eval function?

from easyedit.

run run_knowedit_llama2.py with Llama-2-7b & Wiki_recent_test

Traceback (most recent call last):
File "EasyEdit/examples/run_knowedit_llama2.py", line 208, in
metrics, edited_model, _ = editor.edit(
File "EasyEdit/easyeditor/editors/editor.py", line 171, in edit
return self.edit_requests(requests, sequential_edit, verbose, **kwargs)
File "EasyEdit/easyeditor/editors/editor.py", line 366, in edit_requests
summary_metrics(all_metrics)
File "EasyEdit/easyeditor/editors/utils.py", line 33, in summary_metrics
[metric[eval][key][lkey] for metric in all_metrics])
File "EasyEdit/easyeditor/editors/utils.py", line 33, in
[metric[eval][key][lkey] for metric in all_metrics])
KeyError: 'Subject_Aliasing_acc'

from easyedit.

def summary_metrics(all_metrics):
    if isinstance(all_metrics, dict):
        all_metrics = [all_metrics, ]
    logs_dir = './logs'
    if not os.path.exists(logs_dir):
        os.makedirs(logs_dir)
    output_file = os.path.join(logs_dir, 'results.json')
    with open(output_file, 'w') as f:
        json.dump(all_metrics, f, ensure_ascii=False, indent=4)

    mean_metrics = dict()
    for eval in ["pre", "post"]:
        mean_metrics[eval] = dict()
        for key in ["rewrite_acc", "rephrase_acc"]:
            if key in all_metrics[0][eval].keys():
                mean_metrics[eval][key] = np.mean([metric[eval][key] for metric in all_metrics])
        for key in ["locality", "portability"]:
            if key in all_metrics[0][eval].keys() and all_metrics[0][eval][key] != {}:
                mean_metrics[eval][key] = dict()
                for lkey in all_metrics[0][eval][key].keys():
                    if lkey.endswith("acc"):
                        mean_metrics[eval][key][lkey] = np.mean(
                            [metric[eval][key][lkey] for metric in all_metrics])
    # mean_metrics["time"] = np.mean([metric["time"] for metric in all_metrics])

    print("Metrics Summary: ", mean_metrics)

from easyedit.

Hello, Subject_Aliasing_acc is a metric under portability, and not all edited data have this. However, locality evaluates the model's locality, and every data point will have it. Our evaluation script has already handled cases where Subject_Aliasing_acc is missing. Could you please provide your eval function?

Looking forward to your reply

from easyedit.

Thanks for fixing this bug with your updated code.

from easyedit.