RuntimeError: "addmm_impl_cpu_" not implemented for 'Half' about mergekit HOT 1 CLOSED

This error is occurring because your inference code is trying to run the model in fp16 on the CPU, which transformers/pytorch do not support. If you have a GPU on the machine you're running this on, I'd recommend using it - you can do that by changing model_kwargs like so:

    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": False, "device_map": "auto"},

If you don't have a GPU then you might be able to run the model in either fp32 or bf16 precision. Try like so:

    model_kwargs={"torch_dtype": torch.float32, "load_in_4bit": False}, # or torch.bfloat16

from mergekit.