Negative Sinkhorn divergences about geomloss HOT 2 OPEN

Hi @johannbrehmer ,

Thanks a lot for your detailed report!
A quick answer would be to tell you that this (unfortunate) negative output of the SamplesLoss layer is due to the default value of the scaling parameter that I agressively set to 0.5. As discussed in this tutorial, using a more conservative value of 0.7 or 0.9 should allow you to spend more time in the multiscale Sinkhorn loop (an annealing descent), and thus converge closer to the actual value of the (de-biased) Sinkhorn loss. In your case, something like 0.038 > 0.

Nevertheless, since your example is one of the first that I encounter where changing the scaling parameter actually has a meaningful impact, I played around with it for a bit. Surprise: when we display it in 3D, we observe a few outliers lying far away from the unit sphere.

If we remove these extreme points with a code along the lines of:

x2 = x1.clone()
print(x2[(x1**2).sum(-1) > 1.5, :])
x2[(x1**2).sum(-1) > 1.5, :] = 0

print(sinkhorn(x0, x2).item())

We immediately retrieve a positive value even when scaling = 0.5. As far as I can tell, it thus looks like your example is a perfect worst-case scenario for the multiscale Sinkhorn loop implemented by GeomLoss. This is in line with the theoretical understanding of the algorithm that we're trying to develop with @bernhard-schmitzer : your configuration is pretty close to the "lone wolf" scenario discussed at Example 1, page 17 and Remark 9, page 21 of this important reference paper.

I'm pretty happy to see these theoretical considerations come to life in a real use-case :-)
If you have any other question, feel free to ask; otherwise, I'll let you close this issue.
Best regards,

Jean

P.S.: Out of curiosity, may I ask why you're interested in OT theory? You seem to work mostly on high-energy physics, and I'd be delighted to know how our codes may be of any use to you!

from geomloss.

Hi Jean,

Thanks a lot for the fast and detailed answer! I'll have a closer look at the references you gave, but this makes sense to me.

I arrived at this worst-case scenario not randomly, but by training a generative model on minimizing the Sinkhorn divergence between generated samples and some training data. I had some anomalous results, now I hope that changing the scaling parameter might stabilize my training.

Thanks again,
Johann

PS: Indeed, my background is in high-energy physics, but I am now interested in probabilistic and generative ML models... hence OT. But I don't know much about the theory at all.

from geomloss.