ssbuild / aigc_zoo Goto Github PK
View Code? Open in Web Editor NEWaigc model for deep training
License: Apache License 2.0
aigc model for deep training
License: Apache License 2.0
aigc_zoo/src/model_zoo/chatglm2/chatglm_model.py
Lines 51 to 56 in 1d2294a
这边应该和chatglm_finetuning保持一致?如果实在单轮对话的情况,也会变成Round 1: 问:答:的形式,与训练时的样式不匹配
以下是部分日志
Traceback (most recent call last):
File "/home/darkness/LLM/aigc_serving/serving/model_handler/base/infer.py", line 214, in worker_ds
self.init_model()
File "/home/darkness/LLM/aigc_serving/serving/model_handler/base/infer.py", line 147, in init_model
self.model, self.config, self.tokenizer = call_method(device_id)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/LLM/aigc_serving/serving/model_handler/bluelm/infer.py", line 37, in _load_model
tokenizer, config, _, _ = dataHelper.load_tokenizer_and_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/deep_training/data_helper/data_helper.py", line 190, in load_tokenizer_and_config
tokenizer = load_tokenizer(tokenizer_name=tokenizer_name or model_args.tokenizer_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/numpy_io/pytorch_loader/tokenizer_config_helper.py", line 41, in load_tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, **tokenizer_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 768, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2024, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 76, in init
super().init(
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 367, in init
self._add_tokens(
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 467, in _add_tokens
current_vocab = self.get_vocab().copy()
^^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 110, in get_vocab
vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
^^^^^^^^^^^^^^^
File "/home/darkness/miniconda3/envs/aigc_serving/lib/python3.11/site-packages/aigc_zoo/model_zoo/bluelm/tokenization_bluelm.py", line 106, in vocab_size
return self.sp_model.get_piece_size()
^^^^^^^^^^^^^
AttributeError: 'BlueLMTokenizer' object has no attribute 'sp_model'
INFO:root:'BlueLMTokenizer' object has no attribute 'sp_model'
建议模仿 paddlenlp, 把deep_training 和 aigc_zoo 合并,这样也可以少管理一个库的版本
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.