Official Support for GGUF Quantization in BigCode Starcoder2 to Enhance Accessibility and Efficiency about starcoder2 HOT 4 CLOSED

Hi, StarCoder2 is now integrated in llama.cpp: ggerganov/llama.cpp#5795
(from: https://x.com/sourab_m/status/1764583142499954942?s=20)

from starcoder2.

awful random users sweating to make some ugly cross compatibility that you owners of the project should be taking care about and including it in the release.

I have to disagree, things implemented by the community are in many cases much better than what the authors can come up with, hence the great power of open-source 🙂

And FYI the llama.cpp integration is perfectly functional and was done by an HF employee, you can use it in Ollama as mentioned in the tweet.

from starcoder2.

Hi there, sorry but I'm not sure how this is related to my request. This is exactly what I am complaining about - awful random users sweating to make some ugly cross compatibility that you owners of the project should be taking care about and including it in the release.

How hard is it to start quantizing your own models as well along the release? C'mon 😛

An official implementation by your team would ensure optimal performance and compatibility, eliminating the need for community-driven workarounds.

from starcoder2.

Can't see where the GGUF quantized models are? The hugging face repo does not seem contain them yet? I suggest reading the initial post again. Is it hard to start quantizing your own models from day zero? Why would you want someone else to do that for you?

Open source is awesome but this time it looks more like some lazy effort from the team, specially for a project like starcoder2 that had some veery nice monetary support from Nvidia, ServiceNow and others.

The links you mentioned are more like a pull request thing with a lot of problem solving involved. For the mass end user this is barely useful.

The project is beautiful, useful and very well crafted. Hope to see more people using it.

Thanks.

from starcoder2.