Bug: Cannot quantize a model to BF16 due to an overflow in gguf/quants.py about llama.cpp HOT 5 CLOSED

Before that PR this would be true as long as there are no subnormal weights, after the PR this would always be true.

from llama.cpp.

I can confirm that in fact your PR fixes this issue :)
(and that quantizing a model to f32 and then requantizing to q8_0 produces the same GGUF as quantizing to bf16 and requantizing to q8_0 with this PR, diff says there's no difference in output files and SHA256 is the same - at least in case of Llama3-Instruct-8B)

from llama.cpp.

Try #7843 this is most likely caused by a specific version of numpy (have you updated to 2.x?) and an upcasting side-effect of the conversion.

from llama.cpp.

Yes, i do have numpy 2.0.0 installed, so it's most likely related - i'll check it out, thanks!

from llama.cpp.

one question; do i assume correctly that quantizing a bf16 model to f32, and then requantizing to q8 should result in the same model as the one directly quantized from original bf16 to q8?

from llama.cpp.