Comments (17)
Thanks. Ok, I'm trying to take the code in #12 and update it to the most recent bertviz version. Here's what I got so far (notice the UnicodeDecodeError error when saving the tuned model):
from bertviz import head_view
from transformers import BertTokenizer, BertForSequenceClassification
%%javascript
require.config({
paths: {
d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
}
});
model_path = '/bert/outputs/'
save_model = BertForSequenceClassification.from_pretrained('/bert/outputs/pytorch_model.bin', num_labels=2)
torch.save(save_model.state_dict(), model_path)
from bertviz.transformers_attn import BertModel, BertTokenizer
from bertviz.head_view_bert import show
from bertviz.transformers_attn import BertForSequenceClassification
model_version = 'bert-base-cased'
do_lower_case = True
model_state_dict = torch.load(model_path)
assert model_state_dict is not None
model = BertModel.from_pretrained(model_version, state_dict=model_state_dict,output_attentions=True,num_labels=2)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"
inputs = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', add_special_tokens=True)
token_type_ids = inputs['token_type_ids']
input_ids = inputs['input_ids']
attention = model(input_ids, token_type_ids=token_type_ids)[-1]
input_id_list = input_ids[0].tolist() # Batch index 0
tokens = tokenizer.convert_ids_to_tokens(input_id_list)
head_view(attention, tokens)
Error:
I1125 08:20:57.783921 140693827761984 configuration_utils.py:148] loading configuration file /bert//outputs/paraphrase/pytorch_model.bin
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-5-b4e6db36dc04> in <module>
1 model_path = '/home/rpc/DeepL/Paraphrase/outputs/paraphrase/'
----> 2 save_model = BertForSequenceClassification.from_pretrained('/outputs/pytorch_model.bin', num_labels=2)
3 torch.save(save_model.state_dict(), model_path)
4
5 from bertviz.transformers_attn import BertModel, BertTokenizer
~/.local/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
285 cache_dir=cache_dir, return_unused_kwargs=True,
286 force_download=force_download,
--> 287 **kwargs
288 )
289 else:
~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
152
153 # Load config
--> 154 config = cls.from_json_file(resolved_config_file)
155
156 if hasattr(config, 'pruned_heads'):
~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_json_file(cls, json_file)
184 """Constructs a `BertConfig` from a json file of parameters."""
185 with open(json_file, "r", encoding='utf-8') as reader:
--> 186 text = reader.read()
187 return cls.from_dict(json.loads(text))
188
/usr/lib/python3.6/codecs.py in decode(self, input, final)
319 # decode input (taking the buffer into account)
320 data = self.buffer + input
--> 321 (result, consumed) = self._buffer_decode(data, self.errors, final)
322 # keep undecoded input until the next call
323 self.buffer = data[consumed:]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
from bertviz.
Good to hear! BTW, it's in master now too.
from bertviz.
Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.
from bertviz.
Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.
Hi
Thank you for the quick reply.
yes I did, here is the screenshots
from bertviz.
Have you found a solution? I have a different error when trying to load a BERT fine tuned classification model to analyze via bertviz:
from bertviz.pytorch_transformers_attn import BertForSequenceClassification
from bertviz.pytorch_transformers_attn import BertModel, BertTokenizer
from bertviz.head_view import show
%%javascript
require.config({
paths: {
d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
}
});
import torch
from pytorch_pretrained_bert import BertModel, BertTokenizer, modeling
from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE
from pathlib import Path
save_dir = Path('/home/bert/savedmodelfolder/')
save_dir.mkdir(exist_ok=True)
bert_model = 'bert-base-cased'
save_file = save_dir / modeling.WEIGHTS_NAME
config_file = save_dir / modeling.CONFIG_NAME
model_base = BertModel.from_pretrained(
bert_model,
cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)
)
model_type = 'bert'
model_version = 'bert-base-cased'
do_lower_case = False
# Loading
model_state_dict = torch.load(save_file)
state_dict_with_prefix = {}
for key, value in model_state_dict.items():
state_dict_with_prefix['bert.' + key] = value
model_loaded = BertModel.from_pretrained(save_dir, state_dict=state_dict_with_prefix)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"
show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)
The error is:
AttributeError Traceback (most recent call last)
<ipython-input-15-a0212be47ce3> in <module>
30 sentence_a = "The cat sat on the mat"
31 sentence_b = "The cat lay on the rug"
---> 32 show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)
~/bertviz/head_view.py in show(model, model_type, tokenizer, sentence_a, sentence_b)
57 os.path.join(os.getcwd(), os.path.dirname(__file__)))
58 vis_js = open(os.path.join(__location__, 'head_view.js')).read()
---> 59 attn_data = get_attention(model, model_type, tokenizer, sentence_a, sentence_b)
60 params = {
61 'attention': attn_data,
~/bertviz/attention.py in get_attention(model, model_type, tokenizer, sentence_a, sentence_b, include_queries_and_keys)
57 else:
58 if model_type == 'bert':
---> 59 tokens_a = [tokenizer.cls_token] + tokenizer.tokenize(sentence_a) + [tokenizer.sep_token]
60 tokens_b = tokenizer.tokenize(sentence_b) + [tokenizer.sep_token]
61 token_type_ids = torch.LongTensor([[0] * len(tokens_a) + [1] * len(tokens_b)])
AttributeError: 'BertTokenizer' object has no attribute 'cls_token'
The tokenizer seems to no know what BERT's CLS tokens are... Any ideas? Thank you.
from bertviz.
Hmm, that's an odd one. When I look at the version of BertTokenizer on github, it inherits from tokenization_utils.PretrainedTransformer, which has a cls_token method. Is it possible that somehow the code in either tokenization_bert or tokenization_utils (both in pytorch_transformers_attn library) are out of date? Or somehow being imported from a different path? BTW, I'm working on a change that will decouple bertviz from the transformers library, so this would likely resolve these sorts of issues. I hope to finish it in the next week or two. Thanks.
from bertviz.
Ah yes, I found the problem. Thanks! Everything is beautiful at your end, the paths were incorrect. Thanks again for this tool.
from bertviz.
Thanks for letting me know. BTW, BertViz is now decoupled from transformers for the head and model views, so there shouldn't be a risk of these issues in the future.
from bertviz.
That was fast! Hopefully transformers will stay stable and nothing will break in the future. They tend to keep changing how things work.
I now can visualize my tuned classification model, but interpreting it is hard in my case. It seems that all the attention heads generally attend equally to all words, across all layers. The pattern is always the same as this:
Is there a simple way to inspect the actual attention scores? If not, I'll probably just modify it so that the numbers appear as well.
Again, many many thanks for this.
from bertviz.
Hmm, this is a problem that has happened with other people who loaded fine-tuned models on the previous version of BertViz, e.g., #12 . They were able to resolve this issue but the cause was never clear. I would recommend trying the new version of bertviz in this case. Let me know if any of these solutions work for you.
from bertviz.
Thanks for following up. A couple of thoughts:
- Those additional imports from bertviz.transformers_attn may cause problems downstream, though the error you are getting occurs before that.
- I found some related issues on the Transformers github (both open and closed) where users reported UnicodeDecodeErrors when loading fine-tuned models: https://github.com/huggingface/transformers/issues?utf8=%E2%9C%93&q=unicodedecodeerror . It seems that this error message is a catch-all for different types of problems with model configuration.
Anyways, please let me know if you are able to find a solution there.
from bertviz.
Thanks again for the quick reply. This helped. You were right, the extra calls were necessary and there were problems with paths again. I got it to work, but I'd like to see the "Attention" dropbox menu Sentence_A -> Sentence_B menu. How do I bring it back? Thank you for your patience.
from bertviz.
Hi Rui, I neglected to mention that as part of that change, I removed the functionality to filter the attention, because it was harder to implement when using transformers directly. I do plan to add the functionality back soon, but I don't have an exact timeline unfortunately.
from bertviz.
I see. I found that view quite intuitive (especially when examining longer strings). I'll be on the lookout for it's return. Cheers!
from bertviz.
Hi, I've created a new branch which has the Attention dropdown: https://github.com/jessevig/bertviz/tree/sentence_pair_filter . See head_view_bert.ipynb. Please give it a try!
from bertviz.
Fantastic. This works nicely.
Much appreciated!
from bertviz.
'bert-base-cased'
我也是这个问题 请问是哪个路径的问题?
from bertviz.
Related Issues (20)
- Cannot import bertviz HOT 1
- Is there a simple way to use this code to visualize ViT, DeiT architecture? HOT 1
- How to visualize attention if the sizes of input and output sequence are different? ValueError. HOT 1
- How to use language models such as LLaMA / Alpaca / Vicuna with BertViz? HOT 1
- Will bertviz work for vision transformer?
- cannot import name 'Mapping' from 'collections' HOT 6
- Text truncation
- Takes too much time to run the model_view() visualization
- library installed but not found
- Using [MASK] in a sentence.
- neuron view errors
- Selecting multiple tokens at once.
- Visualize EncoderDecoderModel with tied encoder and decoder
- Any plan on upadating the code for LLaMA models? HOT 11
- Is there any way to "pin" the attention view for a single token? HOT 2
- Request for adding the transformers_neuron_view for LLAMA series models HOT 1
- How to visualize the generated tokens?
- Neuron view
- Bug when visualizing T5 models with generate HOT 2
- Issue visualizing layer attention
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bertviz.