Feature Request: Support Codestral Mamba about llama.cpp HOT 7 OPEN

After we merge #8526 we should try to add full support for this model. cc @compilade

from llama.cpp.

I love the shout-out in the linked blog post!

You can deploy Codestral Mamba using the mistral-inference SDK, which relies on the reference implementations from Mamba’s GitHub repository. The model can also be deployed through TensorRT-LLM. For local inference, keep an eye out for support in llama.cpp. You may download the raw weights from HuggingFace.

That's a really nice nod -- love to see it!

from llama.cpp.

Hey guys, any progress on ETA for it?

from llama.cpp.

#7727 should cover for this model, but with untied embeddings unlike the other Mamba2 models.

from llama.cpp.

FYI, there is an "ngroups" param that changes how layer norm is done : https://github.com/state-spaces/mamba/blob/c0a00bd1808881831ddf43206c69362d4df90cf7/mamba_ssm/modules/mamba2.py#L47

We use ngroups=8. If you forget it or try with ngroups = 1 you'll have a bad time.

Good luck !

from llama.cpp.

I'd love this.

from llama.cpp.

thanks!

from llama.cpp.