Confused about the evaluation of inter-dist metrics. about dialogwae HOT 4 CLOSED

Thanks for your interest!
inter_dist is computed regarding each batch (batch size=1). We want to evaluate how many distinct n-grams the model generates for each context rather than all contexts. #n_samples is always greater than 1 in our evaluation.

We averaged the results for all batches in the test set.

from dialogwae.

Thanks for your reply!
Since DialogWAE samples multiple responses for each context, the inter_dist makes its sense.
However, for other deterministic models, i.e., HRED, only one response is generated for the given context.
Thus, does the inter_dist metric for those models computed regarding all contexts?

from dialogwae.

For deterministic models, we repeated each generated response for n_samples times.
I think it is equivalent to compute regarding all contexts.

from dialogwae.

Thanks for your patient explanations!

from dialogwae.