jinlanfu / gptscore Goto Github PK

In file gpt_inference.py, you calculate the loss via out['logprobs'].
loss = -sum(out['logprobs']["token_logprobs"][i:-1])
However, openai doesn't offer the logprobs in its output of openai.Completion.create().

Could you please tell me how you get the logprobs?

Thanks for your work,

I have a short question:

Why doesn't the Spearman correlation result in the table range from -1 to 1?

GPTScore contains very elaborate experimental results for the generation-based evaluation method for lots of downstream NLG tasks, and thank you so much for your work.

Recently, I also notice that large-scale language models may become a universal and powerful evaluation method, and I also conduct some experimental results on the meta-evaluation benchmarks of dialog generation task, for example, the Empathetic-Eval mentioned in MDD-Eval.

However, I notice that the GPT-3 and other publicly available large-scale language models have a very limited correlation with human judgments (person and spearman scores). I notice that you only conduct the experimental results on the FED-turn and FED-dialog meta-evaluations. I wonder that have you ever noticed the similar experimental results that I found on other meta-evaluation benchmarks (not FED-turn and FED-Dialog).

Looking forward to get response from you.

first of all, thanks for your great work!

As gpt3 instructGPT models are deprecated, im currently updating to gpt-3.5-turbo-instruct
but the problem is log probs and echo parameters are incompatible. i cannot use two parameters at the same time.
how i get the log probs from prompt?

Hi, really appreciate the nice work here! I understand that GPTScore is mainly designed for english text evaluation, I am just wondering, do you think it makes sense to use GPTScore for other languages evaluation, at least for high resource languages such as French and Chinese? Thanks!

Hi, thanks for your brilliant and comprehensive work.

I am recently working on using GPTscore to evaluate the NewsRoom benchmark. I found that there are three different annotations for each sample in the Dataset, which are not very cosistent inherently. If i take all of them for correlation analysis, the results are far below reported.

I would like to know that, how you processed the NewsRoom dataset for correlation analysis (eg. averaging the 3 annotations? or else). Thanks and looking forward for your response.

Great work. I intend to utilize it for my abstractive text summarization paper. Would you be able to upload the related examples? I am interested in evaluating semantic coverage, factuality, informativeness, and fluency following the aspect definitions in your paper.

Kind regards