krystalan / chatgpt_as_nlg_evaluator Goto Github PK

Hi,

could you please share the heuristics you used to convert the free-text responses of chatgpt into a numerical score.
This would increase accessibility / reproducibility and would make comparing different meta evaluation results more consistent.

kind regards.

Hi, could you please kindly provide more details regarding experimental settings? Specifically for SummEval,

• Which model do you use, "gpt-3.5-turbo" or "text-davinci-003" or something else?
• Do you set the temperature to a specific value, or do you use a specific top_p value?
• Does the response from ChatGPT includes explanations, and do you still have these data?
  Thank you and I look forward to your response.

Hi, thanks for your brilliant and comprehensive work.

I am recently working on using chatgpt to evaluate the NewsRoom benchmark since it is reported in this work. I found that there are three different annotations for each sample in the Dataset, which looks not very cosistent.

Since you haven't released the newsroom data, I would like to know that, how you processed the NewsRoom dataset for correlation analysis (eg. averaging the 3 annotations, ultilizing all annotations, or else). Thanks and looking forward for your response.

Thanks for your excellent work!
I have a little question. It is pointed out that the score is from 1 to 5 in bagel dataset instrcution as well as your prompts.
But I found that in bagel json file, the gold informativeness, naturalness and quality often up to 6 or 5.5.
I'm confused. did I miss something?
and could you please provide the source link of bagel? Thank you very much!