Looking for help for using llama.cpp with Phi3 model and LoRA about llama.cpp HOT 7 CLOSED

I find that the structure of Phi2 and Phi3 are named in different way, in Phi2, llama.cpp works fine on transforming LoRA weights to GGML (where the layer is named Wqkv) while in Phi3 this layer is named qkv_proj, I am thinking is this the problem for the failure of llama.cpp on transforming into GGML?

from llama.cpp.

Any update on this? I am running into the same issue. LoRA runs correctly with transformers, but when I convert to llama cpp. It gives me non sense output.

from llama.cpp.

hope this will be fixed as soon as possible

from llama.cpp.

The reason is that llama.cpp treats phi3 as llama architecture, i.e., splitting the merged qkv_proj into q_proj, k_proj and v_proj layers. One way posted by @Raibows at vllm-project/vllm#4715 is to convert the tensor weight of your adapter/lora checkpoint to match it where he gives the script https://gist.github.com/Raibows/079713a060f0c49c8f3b47c227aff722.

I have tested and it is successful for transforming LoRA weights into GGML, but there is another problem that Ollama cannot integrate this GGML LoRA weights back into Phi3-instruct, I think we should somehow merge back the LoRA weights...

from llama.cpp.

anyone?

from llama.cpp.

I have the same issue...

from llama.cpp.

This issue was closed because it has been inactive for 14 days since being marked as stale.

from llama.cpp.