How do I use this tool for my own model? about bertviz HOT 6 CLOSED

Hi, I assume you are still using the Huggingface XLMModel class, but just with your own saved model weights? In that case you can still use the code in https://github.com/jessevig/bertviz/blob/master/head_view_xlm.ipynb but just change following lines, so that model_version is the directory in which your model is saved.

model_version = 'xlm-mlm-ende-1024'
model = XLMModel.from_pretrained(model_version, output_attentions=True)

Just be sure to set output_attentions=True, as above. Does that answer your question?

from bertviz.

No, I am not using huggingface's model. I am running the model found on the facebookresearch repo.

from bertviz.

If that is the case, and you are using the head view or model view, you can still use the notebook above, but you would need to call your model instead of the huggingface one and somehow retrieve the attention weights and reformat them as specified in head_view.py:

def head_view(attention, tokens, sentence_b_start = None, prettify_tokens=True):
    """Render head view
        Args:
            attention: list of ``torch.FloatTensor``(one for each layer) of shape
                ``(batch_size(must be 1), num_heads, sequence_length, sequence_length)``
            tokens: list of tokens
            sentence_b_index: index of first wordpiece in sentence B if input text is sentence pair (optional)
            prettify_tokens: indicates whether to remove special characters in wordpieces, e.g. Ġ
    """

I haven't worked with the FB model so not sure how to implement that on the model end.

from bertviz.

Okay, I'll try that. Thanks for the help!

from bertviz.

The other option is to try to convert your model to a huggingface-compatible model but I'm not sure how easy that would be either. It seems like something others might have done though.

from bertviz.

I will look into that. Thank you!

from bertviz.