ssbuild / aigc_zoo Goto Github PK

这边应该和chatglm_finetuning保持一致？如果实在单轮对话的情况，也会变成Round 1: 问：答：的形式，与训练时的样式不匹配

以下是部分日志
Traceback (most recent call last):
File "/home/darkness/LLM/aigc_serving/serving/model_handler/base/infer.py", line 214, in worker_ds
self.init_model()
File "/home/darkness/LLM/aigc_serving/serving/model_handler/base/infer.py", line 147, in init_model
self.model, self.config, self.tokenizer = call_method(device_id)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/LLM/aigc_serving/serving/model_handler/bluelm/infer.py", line 37, in _load_model
tokenizer, config, _, _ = dataHelper.load_tokenizer_and_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/deep_training/data_helper/data_helper.py", line 190, in load_tokenizer_and_config
tokenizer = load_tokenizer(tokenizer_name=tokenizer_name or model_args.tokenizer_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/numpy_io/pytorch_loader/tokenizer_config_helper.py", line 41, in load_tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, **tokenizer_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 768, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2024, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 76, in init
super().init(
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 367, in init
self._add_tokens(
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 467, in _add_tokens
current_vocab = self.get_vocab().copy()
^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 110, in get_vocab
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 106, in vocab_size
return self.sp_model.get_piece_size()
^^^^^^^^^^^^^
AttributeError: 'BlueLMTokenizer' object has no attribute 'sp_model'
INFO:root:'BlueLMTokenizer' object has no attribute 'sp_model'

建议模仿 paddlenlp, 把deep_training 和 aigc_zoo 合并，这样也可以少管理一个库的版本