Support for Consistency Large Language Models? about llama.cpp HOT 5 OPEN

A 2.4-3.4x speedup, while impressive, cannot be judged in a vacuum. You always need to consider the tradeoffs between speed, quality, and memory use. The difference in memory use should be negligible. However, since you need to specifically finetune the model in order to enable this speedup this degrades the quality of the outputs. The paper reports an increase in perplexity from 8.0 to 9.5. These values are not directly comparable to llama.cpp values since they depend on a lot of factors but I think that this is a lot more than I would intuitively expect from e.g. quantization.

My personal philosophy is that I want to enable the evaluation of large models with negligible precision loss compared to the original weights at the lowest possible cost. So I personally am not interested in implementing something like this.

from llama.cpp.

To clarify my position: I am not willing to do an implementation myself but I would be willing to review someone else's implementation.

from llama.cpp.

My read is the change is to training not code. So it's a fine tuning procedure on the model itself? Is that right or am I missing something?

from llama.cpp.

llama.cpp would still need code changes specifically to support the technique described in the paper.

from llama.cpp.

I'll poke my head over there and see if they're willing to help with that because I'd be very interested in running this. Even with somewhat lower accuracy because I get billed by the millisecond for inference so anything that can speed it up would be a huge savings.

from llama.cpp.