Comments (5)
请问,你有任何新的想法吗?我在源文件中找到了‘PrefixEncoder’的类,似乎被用在了P-TuningV2里
定义在:
65: class PrefixEncoder(torch.nn.Module)
使用在:
736: class ChatGLMModel(ChatGLMPreTrainedModel):
789: def forward():
在这一过程中,官方是将这里的emmbedding作为past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None
送入模型的,不知道能不能帮到你,我也恰好正在研究这里
from chatglm3.
不确定,将跟算法同学进行讨论
from chatglm3.
I'm trying to use GCG with ChatGLM3.
After I read the code carefully, I think generate() actually supports inputs_embeds, which may solve the issue.
I found that input_ids
is only used to provide size to create attention_mask
and position_ids
. When inputs_embeds
is passed in, according to the code
if inputs_embeds is None:
inputs_embeds = self.embedding(input_ids)
the parameter input_ids
does not actually affect the inference results?
So in fact, to use inputs_embeds
as input, we only needmodel(input_ids, inputs_embeds)
Not sure if my understanding is correct?
And I find, when run model(input_ids=input_ids.unsqueeze(0),inputs_embeds=full_embeds)
, the output dimensions of ChatGLM3 seem to be different with those of Llama2 or Vicuna? Need to use something like .permute(1, 0, 2)
?
Not sure about my understanding, thanks a lot in advance for your support!
from chatglm3.
Following the code below does pass embedding as an input, but when using model.generate(), it will prompt an error:"You passed inputs_embeds
to .generate()
, but the model class ChatGLMForConditionalGeneration doesn't have its forwarding implemented. See the GPT2 implementation for an example (huggingface/transformers#21405), and feel free to open a PR with it!"
inputs = tokenizer(MutilTalk_Prompt,padding = 'max_length',max_length = 99)
tensor_input_ids = torch.tensor(inputs['input_ids']+[2])
tensor_input_ids = tensor_input_ids.cuda()
print(tensor_input_ids)
input_embeds = model.transformer.embedding(tensor_input_ids.unsqueeze(0))
outputs = model(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)
#error
outputs = model.generate(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)
from chatglm3.
Following the code below does pass embedding as an input, but when using model.generate(), it will prompt an error:"You passed
inputs_embeds
to.generate()
, but the model class ChatGLMForConditionalGeneration doesn't have its forwarding implemented. See the GPT2 implementation for an example (huggingface/transformers#21405), and feel free to open a PR with it!"inputs = tokenizer(MutilTalk_Prompt,padding = 'max_length',max_length = 99) tensor_input_ids = torch.tensor(inputs['input_ids']+[2]) tensor_input_ids = tensor_input_ids.cuda() print(tensor_input_ids) input_embeds = model.transformer.embedding(tensor_input_ids.unsqueeze(0)) outputs = model(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds) logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True) print(logits_output) #error outputs = model.generate(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds) logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True) print(logits_output)
Oh, I get what u mean now, actually I do not use generate()
, instead I just use model().logits
. And in this case it runs well. But output has a different dimension with that of llama2 or vicuna XD
from chatglm3.
Related Issues (20)
- GLM大模型是Causal Language Model类型的模型么? HOT 1
- 官方的所有openai_api实现的函数调用全部失效了 HOT 8
- 关于微调多轮对话生成的loss_mask格式问题 HOT 1
- ptuning_v2 微调后使用inference_hf.py推理 出现Both `max_new_tokens` (=512) and `max_length`(=8192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation) HOT 4
- openai_api.py 支持并发调用,或者怎么实现并发调用 HOT 1
- 量化加载chatglm3,报错:round_vml_cpu not implemented for Half HOT 5
- 导入huggingface_hub不知道那个版本导入库没有后面的 TextGenerationStreamResponse, Token HOT 9
- [lora finetune] RuntimeError: CUDA error: device-side assert triggered
- AttributeError: 'NoneType' object has no attribute 'peft_type' HOT 3
- api调用出错 HOT 1
- 运行抛异常:CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions HOT 5
- requirements.txt的问题 HOT 1
- problems when finetuning with lora HOT 3
- 【求助】关于算法备案的问题
- 进行p-tuning-v2微调时,报如下错误 HOT 4
- 拼接格式与 chat 接口处理逻辑是否冲突 HOT 2
- lora微调报错 HOT 1
- RMSNorm的不同实现方式
- LORA 微调报错 HOT 1
- bug when running inference_hf.py after finetuning with lora.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chatglm3.