Feature Request: Use Promptfoo to generate DPO datasets about promptfoo HOT 6 CLOSED

Hey @streichsbaer,

#482 starts the work on this. A couple areas where I'm seeking feedback

• I've made chosen and rejected string arrays rather than strings, because some users have multiple columns. Does that make sense to you, or does it make ingesting this data more cumbersome?
• I am assuming that system and question are test case vars. Does that match workflow? I'm wondering if there is a better way to do this more generically. I could just add the vars map to the object.

from promptfoo.

This is an interesting idea, and you're right that we essentially have all the information already. What is the simplest implementation that would be useful to you? e.g. an "export" button or command that outputs {prompt, response, preferred} in a machine-readable format?

from promptfoo.

Glad to hear it!

Yes, an "export" button that produces a file with this format.

[
    {
        "system": "You are an AI assistant...",
        "question": "Generate an approximately...",
        "chosen": "Midsummer House is a moderately...",
        "rejected": " Sure! Here's a sentence that..."
    },
    ...
]

User experience considerations

Evals for 2 models

• Allow setting a thumbs up for an entire column (e.g. auto-choose GPT-4 vs GPT-3.5)
• Allow filtering/exporting for only failed test cases, where it would automatically select the passing one as chosen and the failing one as rejected.

from promptfoo.

Hi @typpo, thanks for that, and sorry for the late response.

• Strings should be fine for chosen and rejected, but having string arrays is ok too. The post-processing of this is not an issue.
• Correct, the system and question are the test case vars. Adding the vars map to the object is a good idea, then there is more flexibility in the post-processing steps.

from promptfoo.

This feature is released in 0.50.0. Thanks and let me know if you have any more feedback!

from promptfoo.

Awesome, thanks @typpo 🙏

from promptfoo.