ricsinaruto / dialog-eval Goto Github PK
View Code? Open in Web Editor NEWEvaluate your dialog model with 17 metrics! (see paper)
Home Page: https://www.aclweb.org/anthology/P19-1567
License: MIT License
Evaluate your dialog model with 17 metrics! (see paper)
Home Page: https://www.aclweb.org/anthology/P19-1567
License: MIT License
Currently every metric is tightly integrated. The framework should be more modular where new metrics can be easily added using a class template and some config parameters.
Currently only fasttext is automatically downloaded, and only SIF average word embedding is used. We could use other word embeddings and sentence representations like BERT.
Currently you can only set which metrics you want computed through the config line. There should be a command line option.
Currently when computing the metrics there are some things that run twice or that are not needed. This should be optimized better.
There should be an option to build vocab entirely from the vector vocab, without text vocab, and more flexible options in general.
Great metrics here: https://arxiv.org/pdf/1810.00671.pdf
After reading through the code:
def update_metrics(self, resp_words, gt_words, source_words):
'''
Params:
:resp_words: Response word list.
:gt_words: Ground truth word list.
:source_words: Source word list.
'''
avg_source = self.avg_embedding(source_words)
avg_resp = self.avg_embedding(resp_words)
# Check for zero vectors and compute cosine similarity.
if np.count_nonzero(avg_resp) and np.count_nonzero(avg_source):
self.metrics['coherence'].append(
1 - distance.cosine(avg_source, avg_resp))
One could import from the code update_metrics
function from CoherenceMetrics
. Although the function receives resp_words
, gt_words
, source_words
, it is not clear to me how to use the function. Could anybody provide a real example of how to call and use Coherence Metrics?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.