the error message:

ImportError: cannot import name 'container_abcs' from 'torch._six' (/opt/conda/lib/python3.8/site-packages/torch/_six.py)

a possible quick-fix would be (cf. https://github.com/NVIDIA/apex/pull/1049/files):

import torch

TORCH_MAJOR = int(torch.__version__.split('.')[0])
TORCH_MINOR = int(torch.__version__.split('.')[1])

if TORCH_MAJOR == 1 and TORCH_MINOR < 8:
    from torch._six import container_abcs
else:
    import collections.abc as container_abcs

Hi,

Thanks for sharing the code

I have been testing your implementation, however I couldn't find corresponding code about AUC curve.

Hello, do you think that you can show one example of using your code for TimeSformer visualization?

Hi,

I am interested to apply this transformer explainability to other huggingface transformer models such as Swin Transformer and CvT. How can I do so?

Thank you.

Hi, i'm wondering how to generater the cam results of a batch of input images at a time to save the time used for visualization. I tried to modify the generate_visualization(original_image) method but failed, thanks for your help!

Hi thank you very much for this tool it is amazing!
I would like to implement it in my own ViT model tough it has been heavily modified. I tried loading the state_dict of my model into your base ViT_LRP one but of course i got some errors due the differences.
Do you advise writing a specific load_state_dict or implementing relprop inside my blocks?
Thank you

Hi, thanks for your excellent work and source code.

I want to visualize my own attention-based architecture, which involves RNN. While implementing this, I find one question: Is it possible to calculate and propagate the relevance of RNN? If the answer is YES, is there any detailed suggestion?

Thanks again for your work and suggestions.

Best wishes

Hi,
i'm wondering how to modify the generate_visualization( ) method in the demo when trying this on CIFAR-100 with img_size=3232 instead of 224224
due to limit of my GPU memory, i failed setting the input size to my vit model as 224

Thanks!

Hi, thank you for the outstanding work!! I have a question on the relprop function of Conv2D.

Based on the formula for LRP in the paper corresponding to the repository you provided in Issue #26, the current implementation appears to be incorrect. This cannot guarantee conservativeness.
The current implementation is as follows:

Such an implementation is equivalent to:

A implementation that I believe is correct is as follows:

Looking forward to your reply!!

Hi thanks for the great work!

I am quite confused about this code segment in the Transformer_explainability.ipynb notebook:

# create heatmap from mask on image
def show_cam_on_image(img, mask):
    heatmap = cv2.applyColorMap(np.uint8(255 * mask), cv2.COLORMAP_JET)
    heatmap = np.float32(heatmap) / 255
    cam = heatmap + np.float32(img)
    cam = cam / np.max(cam)
    return cam

def generate_visualization(original_image, class_index=None):
    transformer_attribution = attribution_generator.generate_LRP(original_image.unsqueeze(0).cuda(), method="transformer_attribution", index=class_index).detach()
    transformer_attribution = transformer_attribution.reshape(1, 1, 14, 14)
    transformer_attribution = torch.nn.functional.interpolate(transformer_attribution, scale_factor=16, mode='bilinear')
    transformer_attribution = transformer_attribution.reshape(224, 224).cuda().data.cpu().numpy()
    transformer_attribution = (transformer_attribution - transformer_attribution.min()) / (transformer_attribution.max() - transformer_attribution.min()) 
    image_transformer_attribution = original_image.permute(1, 2, 0).data.cpu().numpy() # chw -> hwc (rgb)
    image_transformer_attribution = (image_transformer_attribution - image_transformer_attribution.min()) / (image_transformer_attribution.max() - image_transformer_attribution.min())
    vis = show_cam_on_image(image_transformer_attribution, transformer_attribution)
    vis =  np.uint8(255 * vis)
    vis = cv2.cvtColor(np.array(vis), cv2.COLOR_RGB2BGR)
    return vis

image = Image.open('samples/catdog.png')
dog_cat_image = transform(image)

fig, axs = plt.subplots(1, 3)
axs[0].imshow(image);
axs[0].axis('off');

output = model(dog_cat_image.unsqueeze(0).cuda())
print_top_classes(output)

# cat - the predicted class
cat = generate_visualization(dog_cat_image)

# dog 
# generate visualization for class 243: 'bull mastiff'
dog = generate_visualization(dog_cat_image, class_index=243)

axs[1].imshow(cat);
axs[1].axis('off');
axs[2].imshow(dog);
axs[2].axis('off');

as you can see, the image_transformer_attribution is loaded through PIL and transformed by pytorch, so it is in the RGB order.

however, the heatmap is constructed through cv2, so I assume it is in the BGR order? So why we can directly add them together i.e., cam = heatmap + np.float32(img) when they are not in the same color space?

Please correct me if my understanding is wrong here. Thanks!

Most of the NLP tasks use BigBird due to long text support. Can you please extend the class to support BigBird, by adding rules for torch.nn.modules.sparse.Embedding etc?

I am trying to fine tune on a custom image dataset. Using base 224 patch 16, I am setting the all params to false and changing the head output dims to 2 (binary classification). When trying to train this model, I get the error "cannot set a hook on a tensor that does not require grads".

I simply added a conditional statement in vit_lrp.py:

if self.train and x.requires_grad:
    x.register_hook()

if self.train and attn.requires_grad:
    attn.register_hook()

Just making sure this isn't breaking anything as this does fix my issue to fine tune.

Hi,

This work is really interesting. In the paper I saw that you had compared your approach with Grad-CAM among other approaches. However I was unable to find its implementation in the baselines folder. Is there any chance you could add this code?

Thanks in advance

AttributeError: 'ViT' object has no attribute 'relprop'

hugging face:

model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224').cuda()
AttributeError: 'ViTForImageClassification' object has no attribute 'relprop'

Thank you for sharing your project code!
Is that possible to release your ImageNet-Segmentation preparation code?
The public available format is .mat, while the one used in your project is hdf5 and the data is organized in a specific way.
Or any info to obtain the prepared dataset?
Thanks in advance!

Hi @hila-chefer!
I have a question about perturbation results, following your README.md, I try to reproduce the perturbation results.
Is the average of np.mean(num_correct_pertub, axis=1) AUC?
There is no false negative in this experiment, so the y-axis is the classification accuracy, and the x-axis is the different cover rate (0.1~0.9)?
I got a little different result on the rollout method for 55.2 (53.1 in paper), and GradCAM for 41.8 (41.52 in paper) with --vis-class = top.
Are my reproduced results and understanding correct?

Thank you in advance for your help.

Hi Hila,

Thanks for sharing the script to visualize the attention maps.

I am trying to run your DEIT example on custom model (DEIT-base distilled network with 19 classes), but so far have been unsuccessful. I keep getting this error "AttributeError: 'VisionTransformer' object has no attribute 'relprop'"

Here is my saved model weights and biases:
https://drive.switch.ch/index.php/s/dimybgHdzyE90gB

This is how I first load the model from Timm:

basemodel = timm.create_model('deit_base_distilled_patch16_224', pretrained=True, num_classes=19)
model = basemodel

Then I load trained weights and biases from my model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.DataParallel(model)
model.to(device)
criterion = nn.CrossEntropyLoss()
torch.cuda.set_device(0)
model.cuda(0)
criterion = criterion.cuda(0)

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5,weight_decay=3e-5)

PATH = checkpoint_path+'/trained_model.pth'. # Saved model path -- Shared in the link earlier.
checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

When I try to use this model in your example code, I keep getting this error "AttributeError: 'VisionTransformer' object has no attribute 'relprop'"

I was wondering if you would have time to take a look at the uploaded model and see whether you can generate attention map ?

Thanks a lot

Thanks for your great work.
Is there any attempt on integrating CLIP into this model?
I would like to see the attention map of the CLIP model (ViT base)

the problem is similar to issue#31, which is solved.

Hello ,

First, thank you for this great work. It definitely pushes the xai for transformer to a new frontier, and it inspires many people (including me) to work in this area!

I have some question regarding to the evaluation process and I would appreciate your inputs:

1. To evaluate segmentation performance over ImageNet data, we are using the annotated segmentation from Guillaumin et al. 2014 , which consists of 445 imagenet classes of 4276 images. One issue I see is that not all these 445 imagenet classes are part of the standard 1000 classes imagenet dataset (the ones we use to train the VIT or the DEIT ). There are some overlaps, but many classes are either hypernyms/hyponyms/or simply different.

From what I observe , many "errors" in the segmentation are simply because the model misclassfied the image and showed the explanation for the predicted class instead of the GT class. So the explanability error is caused by the model itself instead of the explanability method. Do you think it is useful to separate out cases where the model prediction is incorrect in the first place ? (so we can purely evaluate the XAI method)

I guess this question also applies for other evaluation test we are doing, like pos/neg perturbations. For the segmentation task, we don't have the matching label classes so I understand it is difficult to separate out cases where model made a mistake. However, for other tests (including the ones on NLP tasks), we do have the classification labels. Is there a reason we don't treat misclassification cases separately?

2. For segmentation evaluation, I have a question for the code to calculate average precision:

we first concatenate the (1 - Res) and Res to form a tensor output_AP of size 1x2x224x224.

We then compute AP using output_AP and labels (of shape 1x224x224)

My question is the get_ap_scores:

We iterate every record, as the batch size is 1, each pred has shape 2x224x224 and each tgt has shape 224x224

At line 91, we flattened pred of size (224x224+ 224x224), so the first 224x224 are 1-Res, the second 224x224 are Res,

and we want to compare against a target of size 224x224 + 224x224 where the second 224x224 are the tgt, the first set is 1-tgt

and we use average_precision_score

I think it may not be correct,

it's like having prediction and target to be
[0.1, 0.2, 0.5 0.3] and [0, 0, 1, 1]
then we append the opposite to the 2 lists and compute [0.1, 0.2, 0.5 0.3, 0.9, 0.8, 0.5 0.7] and [0, 0, 1, 1, 1,1 ,0, 0 ] and compute AP...

Maybe the problem is that I got the input tensor shape wrong? I've seen several paper using your evaluation code to produce results
for example, the following work in last year neurips, I can confirm that they use the exact input tensor format as I
described above because I'm able to reproduce their results perfectly. (and the AP looks very wrong)
https://github.com/XianrenYty/Transition_Attention_Maps

Sorry for the long post.. I would appreciate if you can take a look.. I've seen multiple xai works using this exact evaluation code ... and I'm trying to make comparison with my own research,

Thank you

Hi!
When I use for my own ViT model, the following problem has occurred:
torch.nn.modules.module.ModuleAttributeError: 'Linear' object has no attribute 'X'
I don't know why😢Could you please help me? Thank you！

https://arxiv.org/abs/2012.09838

Thank you for the outstanding work!! However, I wonder why you multiply R(A) by G(A) to get the final result. According to my calculation, R(A) is equal to A * G(A) / C, where C is a constant. What would happen if we use R(A) alone? And what’s the motivation to multiply them together?

Looking forward to your reply!!

Hi, thanks for you paper and repo.

I noticed that in bert_pipeline.py you use a different Bert version for each explainability method.

In the above link there is BertForSequenceClassificationTest and BertForClsOrigLrp. Why is this done?

Thank you.

I am adding two lines here from : https://github.com/hila-chefer/Transformer-Explainability/blob/main/BERT_explainability.ipynb

i) expl = explanations.generate_LRP(input_ids=input_ids, attention_mask=attention_mask, start_layer=0)[0]

normalize scores

ii) expl = (expl - expl.min()) / (expl.max() - expl.min())

You normalized the explanation vector score in line ii) here. After the normalization, the most significant one/more got a score of 1.0./-1.0. Other than that, some other tokens may get 0.7 or 0.6, etc. In that case, which tokens are considered as predicted class (i.e., negative sentiment) from the model output? To be specific, did you put a threshold (i.e., >=0.5) for each token to align a specific class?

I was wondering if you would clarify this for me.

Thank you for your interesting work!
I have read your code and get puzzled in add relprop:

        a_sum = a.sum()
        b_sum = b.sum()

        a_fact = safe_divide(a_sum.abs(), a_sum.abs() + b_sum.abs()) * R.sum()
        b_fact = safe_divide(b_sum.abs(), a_sum.abs() + b_sum.abs()) * R.sum()

According normalization in your paper, it should do a.abs() first, then sum them up as a_sum=a.sum(), but you do it reversely in code?

Hi, first thank you for your excellent work, it makes sense and helps me a lot!

But I come across some trouble when trying to fine-tune ViT with another dataset.
I used four 2080ti to train it (torch.nn.DataParallel(model).cuda()), but the memory kept increasing, as if the space taken up after the previous inference and backward operations had not been released.

It is worth noting that single card works well.

There may be problems with the hook-related functions in the code as my training code executed properly on another ViT model with multiple cards.
I'm trying to solve it..., but I think I need your help.

Hi Hila,

Thank you for this brilliant work!

I read your paper and I would like to ask for more details about the raw attention mentioned in your paper.
To my understanding, the raw attention is:
(1) taking the last attention map A^(1)
(2) average according to head, get E_h(A^(1)), which shape is 1ss
(3) choose the row for CLS, get vector 1s
(4) reshape 1s to sqrt(s-1)*sqrt(s-1)
(5) upsampling back to the size of the input image with bilinear interpolation.

I want to confirm with you because I got different visualization of raw attention. I would like to understand if I generate the same version as you did. Thank you very much if you could confirm it.

Besides, I am sorry I did follow the part about applying GradCAM for the transformer, is it then the same as https://github.com/jacobgil/pytorch-grad-cam/blob/master/tutorials/vision_transformers.md ?
If not, could you please tell me how you implement it?

Thank you in advance,
Looking forward to your reply,
Hanwei

While trying to run the file BERT_explainability.ipynb, I am getting this error.

# encode a sentence
text_batch = ["This movie was the best movie I have ever seen! some scenes were ridiculous, but acting was great."]
encoding = tokenizer(text_batch, return_tensors='pt')
input_ids = encoding['input_ids'].to("cuda")
attention_mask = encoding['attention_mask'].to("cuda")

# true class is positive - 1
true_class = 1

# generate an explanation for the input
expl = explanations.generate_LRP(input_ids=input_ids, attention_mask=attention_mask, start_layer=0)[0]
# normalize scores
expl = (expl - expl.min()) / (expl.max() - expl.min())

# get the model classification
output = torch.nn.functional.softmax(model(input_ids=input_ids, attention_mask=attention_mask)[0], dim=-1)
classification = output.argmax(dim=-1).item()
# get class name
class_name = classifications[classification]
# if the classification is negative, higher explanation scores are more negative
# flip for visualization
if class_name == "NEGATIVE":
  expl *= (-1)

tokens = tokenizer.convert_ids_to_tokens(input_ids.flatten())
print([(tokens[i], expl[i].item()) for i in range(len(tokens))])
vis_data_records = [visualization.VisualizationDataRecord(
                                expl,
                                output[0][classification],
                                classification,
                                true_class,
                                true_class,
                                1,       
                                tokens,
                                1)]
visualization.visualize_text(vis_data_records)

Generated Error

NameError Traceback (most recent call last)

/home/cvpr/Transformer-Explainability/BERT_explainability.ipynb Cell 15' in <cell line: 11>()
8 true_class = 1
10 # generate an explanation for the input
---> 11 expl = explanations.generate_LRP(input_ids=input_ids, attention_mask=attention_mask, start_layer=0)[0]
12 # normalize scores
13 expl = (expl - expl.min()) / (expl.max() - expl.min())

File ~/Transformer-Explainability/BERT_explainability/modules/BERT/ExplanationGenerator.py:30, in Generator.generate_LRP(self, input_ids, attention_mask, index, start_layer)
28 def generate_LRP(self, input_ids, attention_mask,
29 index=None, start_layer=11):
---> 30 output = self.model(input_ids=input_ids, attention_mask=attention_mask)[0]
31 kwargs = {"alpha": 1}
33 if index == None:

File ~/miniconda3/envs/maskrcnn/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/Transformer-Explainability/BERT_explainability/modules/BERT/BertForSequenceClassification.py:76, in BertForSequenceClassification.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
73 output = (logits,) + outputs[2:]
74 return ((loss,) + output) if loss is not None else output
---> 76 return SequenceClassifierOutput(
77 loss=loss,
78 logits=logits,
79 hidden_states=outputs.hidden_states,
80 attentions=outputs.attentions,
81 )

NameError: name 'SequenceClassifierOutput' is not defined

@hila-chefer @shirgur Please look into this issue.

https://github.com/rwightman/pytorch-image-models/blob/f8463b8fa9c0490db093b36acfce71fa2363b8c3/timm/models/vision_transformer.py#L169

I can't found HybridEmbed in your implementation.

Could you suggest how to implement RelProp for HybridEmbed?

I want to use this your visualization method with this ResNetV2 implementation.
https://github.com/jeonsworld/ViT-pytorch/blob/460a162767de1722a014ed2261463dbbc01196b6/models/modeling_resnet.py#L129

Hi @hila-chefer, I have a question about the no_weight_decay function below:

Hello!
I'm very interested in your work!
But I have trouble in running the Perturbation Results in Readme, because of lacking the file!
So, where can I download "imagenet_validation_directory"?
Thank you in advance for your convenience!

Hi @hila-chefer,

First of all, congratulations on your paper, the proposal you present is really great!

I am trying to apply your method as an explicability mechanism to the BERT model that I am using to classify the polarization (sentiment) expressed in a set of tweets. My goal is, through your method, to obtain the parts of the text that led to the prediction made. Basically, I have a BERTForSequenceClassification model, which predicts between three target classes, I also have saved the attention arrays returned by the model using output_attentions = True.

I have reviewed the code from bert_pipeline.py, and also BERT-explainability.ipynb, but I still can't quite figure well out how to apply the code.

I was thinking of using the attentions I got from the trained model, but it seems to me that it would not be possible because in generate_LRP, you make use of: .attention.self.get_attn_gradients () and .attention.self.get_attn_cam (). Right?

In any case, what I would have to do would be to configure the pre-trained model, my tweets, outputs, and with that data and my model, pass all that to generate_LRP, then?

Could you please help me with this?

Hi,

Could you please tell me, if I want to visualize my own ViT model, which pytorch implementation should I use to train my model?

Thanks!

Try to run in Colab the Bert-explainability

! pip install -r requirements.txt
I got the following error
ERROR: Could not find a version that satisfies the requirement opencv_python==3.4.2.17 (from versions: 3.4.0.14, 3.4.8.29, 3.4.9.31, 3.4.9.33, 3.4.10.35, 3.4.10.37, 3.4.11.39, 3.4.11.41, 3.4.11.43, 3.4.11.45, 3.4.13.47, 3.4.14.51, 3.4.14.53, 3.4.15.55, 3.4.16.57, 3.4.16.59, 3.4.17.61, 3.4.17.63, 3.4.18.65, 4.1.2.30, 4.2.0.32, 4.2.0.34, 4.3.0.36, 4.3.0.38, 4.4.0.40, 4.4.0.42, 4.4.0.44, 4.4.0.46, 4.5.1.48, 4.5.2.52, 4.5.2.54, 4.5.3.56, 4.5.4.58, 4.5.4.60, 4.5.5.62, 4.5.5.64, 4.6.0.66)
ERROR: No matching distribution found for opencv_python==3.4.2.17

When run the example cells, I got the persistent error: name 'SequenceClassifierOutput' is not defined

I have not found any difference between vit_base_patch16_224 and deit_base_patch16_224 in ViT_LRP.py? What makes they differerent?

Dear Hila,

Thank you very much for your great wok. I really like it.

Recently, Open AI release a model that embed text and image into a comment space using transformer (https://github.com/openai/CLIP). I am wondering if I can use your work to visilize CLIP model, such that for each word in the sentence, we can see the which area the word pay attentions to?

A example could be found in (https://github.com/jackroos/VL-BERT). In this work, they only show layerwise attention, which is not as good as the one you show in your paper.

Thank you for your help.

Best Wishes,

Alex

Hi, this url https://analyticsindiamag.com/compute-relevancy-of-transformer-networks-via-novel-interpretable-transformer/ may be broken.

Hello there,
Thanks for your nice work and code sharing.

I currently have a need to perform parallel inference on img tensor with {B, C, H, W} shape.

I am now calculating for each category as well as each image individually, and this method is really slow.

Since I am using the calculation results for design loss, the current speed is not acceptable.

I would like to know if you have any suggested speedup method to inference the cam in parallel.

Looking forward to seeing your reply, thx

Hey @hila-chefer ,

Congratulations on this great work

1. Could you pls suggest the evaluation framework for XAI methods for vision transformers?
2. is there any wrapper in the existing code base for other XAI methods like Integrated gradients/Smoothgrad etc?

Best,
@jaiswati

❔Question
Hello,
I think it's a very interestring work! But I want to konw where I can get the dataset in "reproducing results on ViT", for example, "gtsegs_ijcv.mat".
I hope for your help. Thanks!

Hi, Hila, I have a question about relprop in BertModel class in BERT.py:
def relprop(self, cam, **kwargs):
cam = self.pooler.relprop(cam, **kwargs)
# print("111111111111",cam.sum())
cam = self.encoder.relprop(cam, **kwargs)
# print("222222222222222", cam.sum())
# print("conservation: ", cam.sum())
return cam

Why we do not need to reprop the embedding module by self.embeddings.relprop(cam, **Kwargs) here before returning cam?

Hi,

Thanks for the great tool. Say if I have a regression task (predicting some score given texts), I was wondering if it's possible to use this tool if I don't have [CLS] token in my input sequence?

Thank you!

Hello there,

Your work is very nice, thanks for sharing the code 😊

I have been testing your implementation, however I could not make it work with my local models, and I was wondering if someone has some idea or insight that could help me to make it work.

The problem

When I load a locally fine-tuned BERT in BERT explainability example notebook (just changing the name of the community model for the local path of the model), explanations.generate_LRP method returns the following error:

RuntimeError: cannot register a hook on a tensor that doesn't require gradient

I have tested with my local versions of bert-base-uncased and neuralmind/bert-base-portuguese-cased, fine-tuned for multiclass classification.

Complete error output

Running

expl = explanations.generate_LRP(input_ids=input_ids, attention_mask=attention_mask, start_layer=0)[0]

generates the following output:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-49-c92b318867c5> in <module>()
      9 
     10 # generate an explanation for the input
---> 11 expl = explanations.generate_LRP(input_ids=input_ids, attention_mask=attention_mask, start_layer=0)[0]
     12 # normalize scores
     13 expl = (expl - expl.min()) / (expl.max() - expl.min())

16 frames
/content/Transformer-Explainability/BERT_explainability/modules/BERT/ExplanationGenerator.py in generate_LRP(self, input_ids, attention_mask, index, start_layer)
     28     def generate_LRP(self, input_ids, attention_mask,
     29                      index=None, start_layer=11):
---> 30         output = self.model(input_ids=input_ids, attention_mask=attention_mask)[0]
     31         kwargs = {"alpha": 1}
     32 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BertForSequenceClassification.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
     52             output_attentions=output_attentions,
     53             output_hidden_states=output_hidden_states,
---> 54             return_dict=return_dict,
     55         )
     56 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states, return_dict)
    628             output_attentions=output_attentions,
    629             output_hidden_states=output_hidden_states,
--> 630             return_dict=return_dict,
    631         )
    632         sequence_output = encoder_outputs[0]

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states, return_dict)
    131                     hidden_states,
    132                     attention_mask,
--> 133                     layer_head_mask,
    134                 )
    135             else:

/usr/local/lib/python3.7/dist-packages/torch/utils/checkpoint.py in checkpoint(function, *args, **kwargs)
    161         raise ValueError("Unexpected keyword arguments: " + ",".join(arg for arg in kwargs))
    162 
--> 163     return CheckpointFunction.apply(function, preserve, *args)
    164 
    165 

/usr/local/lib/python3.7/dist-packages/torch/utils/checkpoint.py in forward(ctx, run_function, preserve_rng_state, *args)
     72         ctx.save_for_backward(*args)
     73         with torch.no_grad():
---> 74             outputs = run_function(*args)
     75         return outputs
     76 

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in custom_forward(*inputs)
    123                 def create_custom_forward(module):
    124                     def custom_forward(*inputs):
--> 125                         return module(*inputs, output_attentions)
    126 
    127                     return custom_forward

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in forward(self, hidden_states, attention_mask, head_mask, output_attentions)
    507             attention_mask,
    508             head_mask,
--> 509             output_attentions=output_attentions,
    510         )
    511         attention_output = self_attention_outputs[0]

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions)
    232             encoder_hidden_states,
    233             encoder_attention_mask,
--> 234             output_attentions,
    235         )
    236         attention_output = self.output(self_outputs[0], h2)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/content/Transformer-Explainability/BERT_explainability/modules/BERT/BERT.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions)
    346 
    347         self.save_attn(attention_probs)
--> 348         attention_probs.register_hook(self.save_attn_gradients)
    349 
    350         # This is actually dropping out entire tokens to attend to, which might

/usr/local/lib/python3.7/dist-packages/torch/tensor.py in register_hook(self, hook)
    255             return handle_torch_function(Tensor.register_hook, relevant_args, self, hook)
    256         if not self.requires_grad:
--> 257             raise RuntimeError("cannot register a hook on a tensor that "
    258                                "doesn't require gradient")
    259         if self._backward_hooks is None:

RuntimeError: cannot register a hook on a tensor that doesn't require gradient

Steps to reproduce the problem

In BERT explainability example notebook, change the name of the model to the path of a locally fine-tuned model uploaded to Google Drive.

More comments

I am able to load the models directly from HuggingFace, and I can even load the downloaded raw weights of bert-base-portuguese-cased without fine-tuning. It could be a problem with my models' fine-tuning process, however I used standard training scripts, and the models have been working so far. Could it be an incompatibility of library versions? Anyway, I will leave the config.json of those models here:

{
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": true,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2,
    "LABEL_3": 3,
    "LABEL_4": 4
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "problem_type": "single_label_classification",
  "transformers_version": "4.8.1",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

{
  "_name_or_path": "neuralmind/bert-base-portuguese-cased",
  "architectures": [
    "BertForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "gradient_checkpointing": true,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4",
    "5": "LABEL_5",
    "6": "LABEL_6",
    "7": "LABEL_7",
    "8": "LABEL_8",
    "9": "LABEL_9"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2,
    "LABEL_3": 3,
    "LABEL_4": 4,
    "LABEL_5": 5,
    "LABEL_6": 6,
    "LABEL_7": 7,
    "LABEL_8": 8,
    "LABEL_9": 9
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "position_embedding_type": "absolute",
  "problem_type": "single_label_classification",
  "transformers_version": "4.8.1",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 29794
}

Hi,

First, thank you for the great work!
I'm not sure if I missed It or not, but according to your paper, you use normalization on (R_j)^u and (R_k)^v and I could not find it in the code (

I couldn't find normalization in your MatMul module as well (

Is that intentional, or am I missing something?

Thanks a lot!
Best.

Hi there,

Thank you very much for putting this together. It is very cool!

I would like to adapt it to a Bert model that has a linear classifier on top (a class is predicted for each token in the input). I was wondering if you have any tips for the easiest way to do this?

Thanks a lot.

Hey, first of thanks for all the work in this repo!

I'm trying to adjust the sample notebook to work with ViT from the torchvision library,
As you might think already, im getting the "AttributeError: 'VisionTransformer' object has no attribute 'relprop' " error.
As suggested in #30 i should pass/load the weights to the modified implementation of this repo.
I can not figure out how i'm supposed to do this? more specifically: when calling vit_LRP(pretrained=True) how should i pass my own weights/ Torchvision model to be used?

Hello, thank you very much for sharing your work. I have a few questions and look forward to your answers. How do I get the gtsegs_ijcv.mat file? As far as I know, imagenet does not seem to have segmentation labels. Thanks again, and look forward to your reply.

I want to know when to provide a visual template of a target detection model like DETR? thx~