Bug: GGUF of Llama 3 8B appears to use smaug-bpe pretokenizer? about llama.cpp HOT 6 CLOSED

Should the pretokenizer not be llama-bpe?

from llama.cpp.

i have same problem

from llama.cpp.

It appears this happens when the creator uses the wrong tokenizer config:

• https://huggingface.co/failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5-GGUF/discussions/1#665878420a9e4d84afd746b7

You can fix it using this script:

llama.cpp/gguf-py/scripts$ python gguf-new-metadata.py --token_pre llama-bpe input output

from llama.cpp.

Confirmed that the following works with the version I pulled yesterday, and the corrected GGUF will load in current ooba. That still leaves a question of why smaug-bpe was selected for Llama 3 8B instead of llama-bpe. It seems the decision arises in convert-hf-to-gguf.py and there is no apparent option to override that with a command-line option.

python llama.cpp/gguf-py/scripts/gguf-new-metadata.py --pre-tokenizer llama-bpe input_gguf output_gguf

from llama.cpp.

Let me close the issue since it happens on the model's creator side.

from llama.cpp.

Sure. I have a more focused question regarding convert-hf-to-gguf.py and its choice of default pre-tokenizer, but I will explore the script more before asking it.

from llama.cpp.