poloclub / dodrio Goto Github PK
View Code? Open in Web Editor NEWExploring attention weights in transformer-based models with linguistic knowledge.
Home Page: https://poloclub.github.io/dodrio/
License: MIT License
Exploring attention weights in transformer-based models with linguistic knowledge.
Home Page: https://poloclub.github.io/dodrio/
License: MIT License
Hello, authors, It's excellent work! And I have a little bug unsolved when I run dodrio-data-gen.py.
The bug seems here
my_loss,my_logit,attentions=my_model(tokens,attention_mask=masks,labels=labels.long(),output_attentions=True)
The details are shown as follows:
Using device: cuda
Reusing dataset glue (/root/.cache/huggingface/datasets/glue/sst2/1.0.0/7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4)
2. Extracting Attention Weights and Gradients...
Some weights of the model checkpoint at bert-base-uncased were not used when initializing MyBertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing MyBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing MyBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of MyBertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
0%| | 0/127 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/.vscode-server/extensions/ms-python.python-2022.4.1/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
cli.main()
File "/root/.vscode-server/extensions/ms-python.python-2022.4.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "/root/.vscode-server/extensions/ms-python.python-2022.4.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/yukyin/PycharmProjects/test/xiaoyanghua_new/humor/dodrio/data-generation/dodrio-data-gen.py", line 966, in <module>
output = my_model(input_ids=tokens
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 1502, in forward
outputs = self.bert(
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 971, in forward
encoder_outputs = self.encoder(
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 568, in forward
layer_outputs = layer_module(
File "/home/yukyin/anaconda3/envs/exp381/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
TypeError: forward() takes from 2 to 7 positional arguments but 8 were given
I have tried different versions of transformers,but it doesn't work.
As referenced in #4 , I had an issue when trying to visualize a custom dataset that had less entries than the default instanceID=1562
.
This value is defined in several different files.
I suggest making this easier to set, setting it to 1
, or mentioning the need to change this value in dodrio-data-gen.py
and its README.
In src/article/Article.svelte
line 189: <em>semantic and syntractic knowledge contexts</em>.
"syntractic" should be "syntactic"
sorry to disturb, I don't have a clue to open and run it.....could you tell me by which way? pycharm?
I am working on an article on machine translate, needing the dodrio to provide me a visual access to the mechanism during the process of transformer model and I want to change the input into the sentences chosen by myself, but the online demo version seems not able to.
thank you greatly.
Hello,
firstly I want to congratulate you for your brilliant work. I have a question, I am working on text summarization project and my own dataset is in french, is it possible to use Dodrio's visualizations? if so, what needs to be changed?
Thank you
Hi,
I've followed the steps provided in dodrio-data-gen.py
and have generated the JSON files as required.
I have then added them to public/data
and updated src/Main.svelte
to reflect the proper files.
However, when I run npm run dev
, the result is blank, the user interface is all there, but there are no visualizations.
I can see, however, that the instance selection shows elements of the correct dataset.
Any tips on how to fix this?
Using device: cuda
Using the latest cached version of the module from /home/exp-10086/.cache/huggingface/modules/datasets_modules/datasets/glue/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad (last modified on Sat Dec 17 14:32:47 2022) since it couldn't be found locally at glue/glue.py or remotely (ConnectionError).
Reusing dataset glue (/home/exp-10086/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
I think it means the function has a wrong parameter.
How to fix this bug?
Thank you.
Hi, Thanks for the amazing work!
Personally, I focus more on CV field. So, I would like to know if this tool can also be used to visualize Attention in CV.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.