Some bugs of your preprocess code about ubuntu-dialogue-generationv2 HOT 8 CLOSED

negative samples selection have big bias.

You concat one utter's continue sentence as one. So for positive response is average 20 length.
But for negative response is one line in a *.tsv file, that means a single sentence of one utter without concat, its average length is 12 only.

from ubuntu-dialogue-generationv2.

Thanks for finding this issue. Do you have the median lengths, rather than the average lengths? My suspicion is the averages are skewed by a small number of very long utterances, so the bias isn't very big overall (in terms of % of examples)

from ubuntu-dialogue-generationv2.

@ryan-lowe
All Sentence Sum: 15442695
All Sentence Avg: 15.4397305717
All Sentence Std: 13.9343662546
All Sentence Med: 12.0

Neg Sentence Sum: 6408036
Neg Sentence Avg: 12.8059297037
Neg Sentence Std: 11.1376761046
Neg Sentence Med: 10.0

Pos Sentence Sum: 9034659
Pos Sentence Avg: 18.0766932909
Pos Sentence Std: 15.826265086
Pos Sentence Med: 14.0

They are very different.

from ubuntu-dialogue-generationv2.

You can just use the length of the response as a feature.
Test Set
=== response_length_baseline ===
R@1 0.185770194555
R@2 0.321817721091
R@5 0.635695582397

Valid Set
=== response_length_baseline ===
R@1 0.187840494243
R@2 0.33299634934
R@5 0.646307217074

from ubuntu-dialogue-generationv2.

Okay, we'll look into this. It could be fixed short-term by just altering the test set so that it excludes examples where the true response contains several concatenated utterances, but long-term we'll want to generate a new training set as well

from ubuntu-dialogue-generationv2.

@ryan-lowe Thanks.

from ubuntu-dialogue-generationv2.

We have released a new version of the Ubuntu Dialogue Corpus, available here:
https://github.com/rkadlec/ubuntu-ranking-dataset-creator

This should fix the problems regarding the uneven distribution of responses.

from ubuntu-dialogue-generationv2.

@ryan-lowe Thanks!

from ubuntu-dialogue-generationv2.