What happened? Despite <a href="https://huggingface.co/microsoft/p

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Bug: Phi-2 model tokenizer not recognized about llama.cpp HOT 2 OPEN

saeid93 commented on June 30, 2024

Bug: Phi-2 model tokenizer not recognized

from llama.cpp.

Comments (2)

nicolasperez19 commented on June 30, 2024

Can you try saving the phi-2 model without saving the phi-2 tokenizer, like this:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)

model.save_pretrained("./saved_model")

Then try to compile it as you did last time.

from llama.cpp.

saeid93 commented on June 30, 2024

@nicolasperez19 Thank you for your response but how is the model supposed to work without a tokenizer?
I tried the code you mentioned and llamacpp is expecting the tokenizer to be there:

INFO:hf-to-gguf:Loading model: saved_model
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:Set model tokenizer
Traceback (most recent call last):
  File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 2856, in <module>
    main()
  File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 2841, in main
    model_instance.set_vocab()
  File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 116, in set_vocab
    self._set_vocab_gpt2()
  File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 502, in _set_vocab_gpt2
    tokens, toktypes, tokpre = self.get_vocab_base()
  File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 377, in get_vocab_base
    tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
  File "/home/cc/miniconda3/envs/peftenv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 855, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/cc/miniconda3/envs/peftenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2070, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'saved_model'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'saved_model' is the correct path to a directory containing all relevant files for a CodeGenTokenizerFast tokenizer.