Comments (6)
Hi, I assume you are still using the Huggingface XLMModel class, but just with your own saved model weights? In that case you can still use the code in https://github.com/jessevig/bertviz/blob/master/head_view_xlm.ipynb but just change following lines, so that model_version
is the directory in which your model is saved.
model_version = 'xlm-mlm-ende-1024'
model = XLMModel.from_pretrained(model_version, output_attentions=True)
Just be sure to set output_attentions=True, as above. Does that answer your question?
from bertviz.
No, I am not using huggingface's model. I am running the model found on the facebookresearch repo.
from bertviz.
If that is the case, and you are using the head view or model view, you can still use the notebook above, but you would need to call your model instead of the huggingface one and somehow retrieve the attention weights and reformat them as specified in head_view.py:
def head_view(attention, tokens, sentence_b_start = None, prettify_tokens=True):
"""Render head view
Args:
attention: list of ``torch.FloatTensor``(one for each layer) of shape
``(batch_size(must be 1), num_heads, sequence_length, sequence_length)``
tokens: list of tokens
sentence_b_index: index of first wordpiece in sentence B if input text is sentence pair (optional)
prettify_tokens: indicates whether to remove special characters in wordpieces, e.g. Ġ
"""
I haven't worked with the FB model so not sure how to implement that on the model end.
from bertviz.
Okay, I'll try that. Thanks for the help!
from bertviz.
The other option is to try to convert your model to a huggingface-compatible model but I'm not sure how easy that would be either. It seems like something others might have done though.
from bertviz.
I will look into that. Thank you!
from bertviz.
Related Issues (20)
- Is there a simple way to use this code to visualize ViT, DeiT architecture? HOT 1
- How to visualize attention if the sizes of input and output sequence are different? ValueError. HOT 1
- How to use language models such as LLaMA / Alpaca / Vicuna with BertViz? HOT 1
- Will bertviz work for vision transformer?
- cannot import name 'Mapping' from 'collections' HOT 6
- Text truncation
- Takes too much time to run the model_view() visualization
- library installed but not found
- Using [MASK] in a sentence.
- neuron view errors
- Selecting multiple tokens at once.
- Visualize EncoderDecoderModel with tied encoder and decoder
- Any plan on upadating the code for LLaMA models? HOT 11
- Is there any way to "pin" the attention view for a single token? HOT 2
- Request for adding the transformers_neuron_view for LLAMA series models HOT 1
- How to visualize the generated tokens?
- Neuron view
- Bug when visualizing T5 models with generate HOT 2
- Issue visualizing layer attention
- Any Alternative for Vision transformers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bertviz.