jinlanfu / gptscore Goto Github PK
View Code? Open in Web Editor NEWSource Code of Paper "GPTScore: Evaluate as You Desire"
Source Code of Paper "GPTScore: Evaluate as You Desire"
In file gpt_inference.py, you calculate the loss via out['logprobs'].
loss = -sum(out['logprobs']["token_logprobs"][i:-1])
However, openai doesn't offer the logprobs in its output of openai.Completion.create().
Could you please tell me how you get the logprobs?
Thanks for your work,
I have a short question:
Why doesn't the Spearman correlation result in the table range from -1 to 1?
GPTScore contains very elaborate experimental results for the generation-based evaluation method for lots of downstream NLG tasks, and thank you so much for your work.
Recently, I also notice that large-scale language models may become a universal and powerful evaluation method, and I also conduct some experimental results on the meta-evaluation benchmarks of dialog generation task, for example, the Empathetic-Eval mentioned in MDD-Eval.
However, I notice that the GPT-3 and other publicly available large-scale language models have a very limited correlation with human judgments (person and spearman scores). I notice that you only conduct the experimental results on the FED-turn and FED-dialog meta-evaluations. I wonder that have you ever noticed the similar experimental results that I found on other meta-evaluation benchmarks (not FED-turn and FED-Dialog).
Looking forward to get response from you.
first of all, thanks for your great work!
As gpt3 instructGPT models are deprecated, im currently updating to gpt-3.5-turbo-instruct
but the problem is log probs and echo parameters are incompatible. i cannot use two parameters at the same time.
how i get the log probs from prompt?
Hi, really appreciate the nice work here! I understand that GPTScore is mainly designed for english text evaluation, I am just wondering, do you think it makes sense to use GPTScore for other languages evaluation, at least for high resource languages such as French and Chinese? Thanks!
Hi, thanks for your brilliant and comprehensive work.
I am recently working on using GPTscore to evaluate the NewsRoom benchmark. I found that there are three different annotations for each sample in the Dataset, which are not very cosistent inherently. If i take all of them for correlation analysis, the results are far below reported.
I would like to know that, how you processed the NewsRoom dataset for correlation analysis (eg. averaging the 3 annotations? or else). Thanks and looking forward for your response.
Great work. I intend to utilize it for my abstractive text summarization paper. Would you be able to upload the related examples? I am interested in evaluating semantic coverage, factuality, informativeness, and fluency following the aspect definitions in your paper.
Kind regards
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.