ChatML prompt template problems post 2.7.1 about gpt4all HOT 2 CLOSED

This is an upstream issue with OpenHermes-2.5-neural-chat-v3-3-Slerp. It's a merge of OpenHermes-2.5-Mistral-7B and neural-chat-7b-v3-3.

Weyaxi claims you can use either prompt format, but neither the tokenizer_config.json nor the special_tokens_map.json correctly specifies the special tokens required by OpenHermes-2.5-Mistral-7B for ChatML as can be seen here.

Please test with a different model. You can report the issue with this model here, and use the neural-chat-7b-v3-3 prompt template for now.

edit: looks like this problem was already reported and closed as WONTFIX https://huggingface.co/Weyaxi/OpenHermes-2.5-neural-chat-v3-3-Slerp/discussions/5
So unless you want to modify the tokenizer config and convert the model yourself, you can't use ChatML with it. By inspecting the contents of the GGUF I have confirmed that TheBloke did not do this (there are no im_start/im_end tokens in the vocab), despite him repeating the bad recommendation from Weyaxi to use ChatML.

from gpt4all.

Ok thank you very much. It is a pity, because it is a great model. Looks like it is hard to support, if people try to use ChatML because of a bad recommendation in the model card and then realize it will not work.

from gpt4all.