Comments (8)
negative samples selection have big bias.
You concat one utter's continue sentence as one. So for positive response is average 20 length.
But for negative response is one line in a *.tsv file, that means a single sentence of one utter without concat, its average length is 12 only.
from ubuntu-dialogue-generationv2.
Thanks for finding this issue. Do you have the median lengths, rather than the average lengths? My suspicion is the averages are skewed by a small number of very long utterances, so the bias isn't very big overall (in terms of % of examples)
from ubuntu-dialogue-generationv2.
@ryan-lowe
All Sentence Sum: 15442695
All Sentence Avg: 15.4397305717
All Sentence Std: 13.9343662546
All Sentence Med: 12.0
Neg Sentence Sum: 6408036
Neg Sentence Avg: 12.8059297037
Neg Sentence Std: 11.1376761046
Neg Sentence Med: 10.0
Pos Sentence Sum: 9034659
Pos Sentence Avg: 18.0766932909
Pos Sentence Std: 15.826265086
Pos Sentence Med: 14.0
They are very different.
from ubuntu-dialogue-generationv2.
You can just use the length of the response as a feature.
Test Set
=== response_length_baseline ===
R@1 0.185770194555
R@2 0.321817721091
R@5 0.635695582397
Valid Set
=== response_length_baseline ===
R@1 0.187840494243
R@2 0.33299634934
R@5 0.646307217074
from ubuntu-dialogue-generationv2.
Okay, we'll look into this. It could be fixed short-term by just altering the test set so that it excludes examples where the true response contains several concatenated utterances, but long-term we'll want to generate a new training set as well
from ubuntu-dialogue-generationv2.
@ryan-lowe Thanks.
from ubuntu-dialogue-generationv2.
We have released a new version of the Ubuntu Dialogue Corpus, available here:
https://github.com/rkadlec/ubuntu-ranking-dataset-creator
This should fix the problems regarding the uneven distribution of responses.
from ubuntu-dialogue-generationv2.
@ryan-lowe Thanks!
from ubuntu-dialogue-generationv2.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ubuntu-dialogue-generationv2.