Strange prompt(s) about alpaca_eval HOT 5 CLOSED

1. Thanks for pointing this issue out! Definitely agree that this format looks weird. Most of the OASST instructions don't look like this, so it could be an exporting issue for the few that are affected. We will look more into this (but given that only 12 are affected, it shouldn't change the final win-rates too much).

2. Can you clarify what you mean use the prompt template as instructions? The prompt template for each model is provided in the respective model_configs directory.

from alpaca_eval.

This prompt is applied rather often
https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/text_davinci_003/prompt.txt

I think davinci 3 does not need any template at all, the input can just be an instruction. The same is true for many other models, and I think this may be hurting the Cohere model in particular.

from alpaca_eval.

So the current prompt works well for Davinci003, in the sense that it doesn't make a mistake in understanding the formatting. If you have a suggestion to update the Cohere model template, please submit a PR with the updated config and results and we’d be happy to incorporate it.

from alpaca_eval.

Hi @sanderland quick follow-up saying that we went through the Cohere prompt engineering page when making the prompt and we didn't see any information about a special template, which is why we originally used a simple one like davinci-003. Other models have specific prompt templates, e.g. Claude. Let us know if we missed the Cohere prompt template!

from alpaca_eval.

Hey @YannDubs

• Command is an instruction finetuned model, and when using it with instructions, it indeed does not need any template.
  • I think this is also true for davinci-003 and most instruction finetuned models in general.
  • The template you used may induce shorter answers for command in particular.
• The Client.chat method in the sdk will automatically add a template based suitable for inducing a more conversational style.
  • However, it is not clear what this benchmark is primarily testing (instruct models vs conversational ones being a little different)

from alpaca_eval.