Average Increase/Drop Evaluation about relevance-cam HOT 4 OPEN

Thank you for your interest!

below is the code for Average Drop and Average Increase.

if you have other question, feel free to ask me :)

for original_image in dataset:

rele_sort = np.sort(relevance_cam.reshape(-1))
relevance_cam = relevance_cam * (relevance_cam > rele_sort[-int(224*224*0.5)])  ## top 50% pixels

rele_cam_in = Variable(preprocess_image(img * (relevance_cam[...,np.newaxis]))).cuda()

y = softmax(model(original_image))
rele_o = softmax(model(rele_cam_in))

weight_dict = dict()
weight_dict['y'] = list()
weight_dict['relevance'] = list()

weight_dict['y'].append(output[:,maxindex].detach().cpu().numpy()[0])
weight_dict['relevance'].append(rele_o[:,maxindex].detach().cpu().numpy()[0])

y = np.array(weight_dict['y'])
relevance_dict = np.array(weight_dict['relevance'])

posit_r = (y - relevance_dict) > 0

relevance_aver = (y - relevance_dict) * posit_r / y

r_denom = relevance_aver > 0

print("Average Drop:", np.sum(relevance_aver) * 100 / r_denom.sum())

print("Average Increase:",1-posit_r.mean())

from relevance-cam.

Thank you for the code! I'm having a little trouble reproducing the metrics, and wanted to clarify my understanding: according to the paper, "only the cases which the labels and the model predictions match are considered to measure the contribution exactly." Does this mean that only images where argmax(y) == true_label are candidates for A.D. and A.I?

Also, could you clarify relevance_cam setup: is it identical to how R_CAM is produced in Multi_CAM.py?

Really appreciate the help!

from relevance-cam.

yes, you are right @dash102.

The sentence means images where argmax(y) == true_label are candidates for A.D. and A.I

and relevance_cam is R_CAM in Multi_CAM.py

Thanks

from relevance-cam.

Hi @mongeoroo ,

Thank you for the clarifications! I have my best attempt at a working file that reproduces the evaluation here, but have not been able to replicate the paper's results yet. It may be that I haven't yet hit the right random set of 2000 images, but I also had a few small things I thought may be different to match the paper's formulas:

• On line 107, I tried masking the normalized image instead of normalizing the masked image, so that the zero-mask did not affect the RGB image's statistics.
• On line 130, I divided by the number of valid candidates out of 2000 attempted images, as that seemed to reflect the N in formula 10.

I tried the provided code (normalizing the masked image and dividing by r_denom.sum()) as well as these changes but am getting numbers different from the table, so I might be totally missing something in my attempt. Could you help point me in the right direction for how this code might be computing A.D. and A.I. incorrectly? I would love to help contribute a PyTorch-ified implementation of this evaluation script to the repository if the code ends up working out!

from relevance-cam.