Qwen2 GPTQ break in cpp_model.Model.np_bestla_qpack about neural-speed HOT 1 CLOSED

@yuchen2580 Hi, thanks for your issue.

I noticed this problem for Qwen1.5-0.5B-Chat-GPTQ-Int4.

I think this GPTQ model has some problems probably. It should be from https://hf-mirror.com/Qwen/Qwen1.5-0.5B.

In Qwen1.5-0.5B-Chat-GPTQ-Int4, there is no lm_head.weight in the model.safetensor. But original Qwen1.5-0.5B, it has this weight.

if you use these commands to check them, you will see this weird problem.

from safetensors.torch import load_file
tensors = load_file("model.safetensors")
tensors.keys()

Qwen1.5-0.5B-Chat-GPTQ-Int4:

Qwen1.5-0.5B:

Qwen1.5-0.5B-Chat-GPTQ-Int4 also doesn't work on the HF, which means I suspect there is something wrong with this model.

Another way, you can try https://huggingface.co/Qwen/Qwen1.5-7B-Chat-GPTQ-Int4. it works.

from neural-speed.