[BUG] Different Outputs Streaming (More Coherent) vs Single Message (Writes Transcript) about oobabot HOT 8 OPEN

Awesome, thanks for the update on this!

It's interesting that there's a difference... I wonder if you switch back to "single message" mode whether you would continue to get the improved behavior or if it would regress.

The main difference might be that any of the settings which split the message into parts might give the bot more context around what a chat transcript should look like, so it's easier for it to get the gist of what we want it to generate.

Let me know if the other investigation turns up anything, and thanks for the help!

from oobabot.

How would I try this?

from oobabot.

I get different responses even though I have the parameters identical in config.yml for oobabot and in text-gen-webui. I tried adding the new min_p parameter to config.yml and it loads but I don't know if its working. The params are
request_params:
max_new_tokens: 4000
do_sample: true
temperature: 1.6
top_p: 1
typical_p: 1
epsilon_cutoff: 0
eta_cutoff: 0
tfs: 1
top_a: 0
repetition_penalty: 1.18
min_p: 0.26
top_k: 20
min_length: 0
no_repeat_ngram_size: 0
num_beams: 1
penalty_alpha: 0
length_penalty: 1
early_stopping: false
mirostat_mode: 0

In text-gen-webui I ask "how can I increase my power level past 9000?" and it gives me a list of things to do but in discord it just says "become one with the force" no matter what settings I set?

from oobabot.

could be a difference in prompting. The webui's selection and oobabot's preset system prompt are both very different.

from oobabot.

I am wondering that as well, does oobabot inherit the chat-instruction template from text-gen? the character context for text-gen and the oobabot personality are both "The following is a conversation with an AI Large Language Model. The AI has been trained to answer questions, provide recommendations, and help with decision making. The AI follows user requests. The AI thinks outside the box."

from oobabot.

Don't forget the instruction format, including tags like [INST] and <> if using a llama2-chat model. These aren't included by default in oobabot.

from oobabot.

I'm using dolphin-mixtral and the model card says this about the prompt format
Prompt format: This model uses ChatML prompt format.

<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

So in the persona I have <|im_start|>system You are a discord AI bot. The AI follows instructions and is helpful. <|im_end|>

The model is still is wonky. right now it spams emojis with every reply. The temp is at 1 in the config but it doesn't seem to change anything?

from oobabot.

I'm using dolphin-mixtral and the model card says this about the prompt format Prompt format: This model uses ChatML prompt format.

<|im_start|>system You are Dolphin, a helpful AI assistant.<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant

So in the persona I have <|im_start|>system You are a discord AI bot. The AI follows instructions and is helpful. <|im_end|>

The model is still is wonky. right now it spams emojis with every reply. The temp is at 1 in the config but it doesn't seem to change anything?

I am having similar issues, did you ever find anything out?

from oobabot.