reppy4620 / dialog Goto Github PK
View Code? Open in Web Editor NEWA PyTorch Implementation of japanese chatbot using BERT and Transformer's decoder
License: MIT License
A PyTorch Implementation of japanese chatbot using BERT and Transformer's decoder
License: MIT License
Dialog training has just finished.
Then, the training log is as follows:
INFO:root:*** Initializing ***
INFO:root:Preparing training data
INFO:root:Define Models
INFO:root:Define Loss and Optimizer
INFO:root:Start Training
Epoch: 1: 100%|██████████| 54882/54882 [9:39:27<00:00, 1.58it/s, Loss: 2.18444]
Epoch: 2: 100%|██████████| 54882/54882 [9:37:28<00:00, 1.58it/s, Loss: 2.54174]
Epoch: 3: 100%|██████████| 54882/54882 [9:31:07<00:00, 1.60it/s, Loss: 2.34356]
*** Saved Model ***
おはようございます
*** Saved Model ***
おはようございます。
*** Saved Model ***
おはようございます
Then I tried to test the trained 'ckpt.pth' file using run_eval.py.
But, the responses of the trained file is as follows:
$ python3 run_eval.py
You>おはよう
BOT>おはようございます
You>今日は疲れた
BOT>おはようございます
You>美味しいものを食べたい
BOT>おはようございます
Responses are just only 'おはようございます'.
What's wrong?
Please let me know your thinking.
I tried to use the evaluation edition by downloading pretrained weight.
But, when I entered the command "python3 run_eval.py", an error occurred below,
Traceback (most recent call last):
File "run_eval.py", line 12, in
state_dict = torch.load(f'{Config.data_dir}/{Config.fn}.pth')
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/serialization.py", line 765, in _legacy_load
result = unpickler.load()
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/serialization.py", line 721, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/serialization.py", line 174, in default_restore_location
result = fn(storage, location)
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/serialization.py", line 154, in _cuda_deserialize
return storage_type(obj.size())
File "/home/m-ishihara/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 480, in _lazy_new
return super(_CudaBase, cls).new(cls, *args, **kwargs)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 1.96 GiB total capacity; 1.40 GiB already allocated; 11.81 MiB free; 1.48 GiB reserved in total by PyTorch)
After investigating gpu memory usage, I found all the gpu memory was free.
Please tell me what's wrong with it.
notebookのDialog-Evaluation.ipynbを実行したところ、
!python run_eval.py
のセルでエラーが発生しました。
Traceback (most recent call last):
File "run_eval.py", line 15, in <module>
tokenizer = Tokenizer.from_pretrained(Config.model_name)
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1591, in from_pretrained
list(cls.vocab_files_names.values()),
OSError: Model name 'bert-base-japanese-whole-word-masking' was not found in tokenizers model name list (cl-tohoku/bert-base-japanese, cl-tohoku/bert-base-japanese-whole-word-masking, cl-tohoku/bert-base-japanese-char, cl-tohoku/bert-base-japanese-char-whole-word-masking). We assumed 'bert-base-japanese-whole-word-masking' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
config.pyのmodel_nameを'cl-tohoku/bert-base-japanese-whole-word-masking'として追加でライブラリをインストールすればこのエラーは発生しなくなるのですが、次はモデル読み込みでレイヤー名が合わずエラーとなってしまいます。
Traceback (most recent call last):
File "run_eval.py", line 21, in <module>
model.load_state_dict(state_dict['model'])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for EncoderDecoder:
Missing key(s) in state_dict: "encoder.embeddings.position_id
ちなみに、現時点でpipでインストールされるライブラリのVersionは下記となっております。
emoji-0.6.0 neologdn-0.4 sacremoses-0.0.43 sentencepiece-0.1.94 tokenizers-0.9.2 transformers-3.4.0
お手数をおかけしますが、解決方法をご教授ください。
I attempted to run the code by executing !python main.py, and unfortunately, I encountered an issue with the following error:
!python main.py
Traceback (most recent call last):
File "/content/Dialog/main.py", line 10, in
from nn import build_model
File "/content/Dialog/nn/init.py", line 2, in
from .model import EncoderDecoder, build_model
File "/content/Dialog/nn/model/init.py", line 1, in
from .encoder_decoder import EncoderDecoder, build_model
File "/content/Dialog/nn/model/encoder_decoder.py", line 5, in
from .encoder import build_encoder
File "/content/Dialog/nn/model/encoder.py", line 2, in
from transformers.modeling_bert import BertModel
ModuleNotFoundError: No module named 'transformers.modeling_bert'
Additionally, when attempting to download the file with files.download('./data/ckpt.pth'), I received the following error:
from google.colab import files
files.download('./data/ckpt.pth')
FileNotFoundError Traceback (most recent call last)
in <cell line: 2>()
1 from google.colab import files
----> 2 files.download('./data/ckpt.pth')
/usr/local/lib/python3.10/dist-packages/google/colab/files.py in download(filename)
223 if not _os.path.exists(filename):
224 msg = 'Cannot find file: {}'.format(filename)
--> 225 raise FileNotFoundError(msg) # pylint: disable=undefined-variable
226
227 comm_manager = _IPython.get_ipython().kernel.comm_manager
FileNotFoundError: Cannot find file: ./data/ckpt.pth
I believe that the first error is related to a missing module, but I'm uncertain how to resolve it. Additionally, the second error seems to indicate that the file ckpt.pth cannot be found in the specified directory.
I would greatly appreciate it if you could provide some guidance on how to address these issues.
Hello ,
when I execute the code, I get the following error: 'Tokenizer' object has no atribute 'ids_to_tokens
, I don't know where it comes from. Thanks you
Thank you very much for the excellent program you have created.
I have trained 5 epochs in exactly the same way, but all the bots return "Good morning".
It seems to be a simple mistake, but I would appreciate any feedback.
In addition, I have modified the following part of the program so that it will work on 7/29/2021.
1.config.py
#model_name = "bert-base-japanese-whole-word-masking"
model_name = "cl-tohoku/bert-base-japanese-whole-word-masking"
2.tokenizer.py
#from transformers.tokenization_bert_japanese import BertJapaneseTokenizer
from transformers import BertJapaneseTokenizer
Sincerely yours.
The following is what I did during the training.
python main.py
INFO:root:*** Initializing ***
INFO:root:Preparing training data
INFO:root:Define Models
Some weights of the model checkpoint at cl-tohoku/bert-base-japanese-whole-word-masking were not used when initializing BertEncoder: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias']
The following is errors of my chat bots
ckpt_4 (epoch 5)
You>こんにちは。
BOT>おはようございます
You>暑いですね。
BOT>おはようございます
You>元気ですか?
BOT>おはようございます
You>今日雨フルらしいよ
BOT>おはようございます
ckpt (original model)
You>おはよう
BOT>おはようございます
You>元気ですか?
BOT>おはようございます
You>むむ
BOT>おはようございます
You>
Hello,
thank you for providing such an interesting project!
I have a question about the following lines:
Dialog/nn/model/encoder_decoder.py
Lines 52 to 53 in b33fa57
I do not understand why the two source embeddings are concatenated.
What does this implementation mean?
thanks!
Thank you for your support.
BTW, when I tried to run using a pre-trained model. unfortunately, I got the error below:
python3 run_eval.py
Traceback (most recent call last):
File "run_eval.py", line 17, in
model.load_state_dict(state_dict['model'])
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for EncoderDecoder:
Missing key(s) in state_dict: "encoder.embeddings.position_ids".
Please let me know if you have any idea.
Thank you for the great work.
I followed the usage guide in the github but some errors occurred. Any code not changed.
The following error occurred when copying pretrained model to './pretrained' folder and executing 'python main.py'.
"OSError: file ./pretrained/config.json not found"
'config.json' file was not found in your github. When I copied the 'config.json' from original Bert, it was cleared.
After that, I got an error "OSError: file ./pretrained/pytorch_model.bin not found"
Please give me some tips to solve it.
Thanks in advance.
Seungkwon.
I added another Japanese around 2000 lines data to train more.
Also I added voice reply functionality.
https://jweb.asia/26-it/ai/51-bert-chatbot.html
I was trying to test the pretrained model.
But an error popped out.
the version of my packages:
transformers: 3.3.1
torch: 1.3.1
Can I ask the exact version of the transformer and torch you used for this project?
Thank you so much in advance.
RuntimeError Traceback (most recent call last)
in
3 tokenizer = Tokenizer.from_pretrained(Config.model_name)
4 model = build_model(Config).to(device)
----> 5 model.load_state_dict(state_dict['model'])
6 model.eval()
7 model.freeze()
~/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
837 if len(error_msgs) > 0:
838 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 839 self.class.name, "\n\t".join(error_msgs)))
840 return _IncompatibleKeys(missing_keys, unexpected_keys)
841
RuntimeError: Error(s) in loading state_dict for EncoderDecoder:
Missing key(s) in state_dict: "encoder.embeddings.position_ids".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.