Re: Self-similarity hook about diode HOT 3 CLOSED

@animesh-007 Here is the code for SelfSimilarityHook

There are no relevant papers because its just the cosine distance between vectors 😃
source: https://scikit-learn.org/stable/modules/metrics.html#cosine-similarity

If we can represent inputs (e.g images) as vectors (e.g x \in \mathcal{R}^n}) that contain semantically relevant features for those inputs, cosine distance tells us how close they are on the unit circle i.e the angle between the 2 vectors, don't care about the magnitude.

We tried to use it to reduce the similarity between the batch of images that are being optimized (with very little success), so that each image in the batch has different colors and textures. We saw that a batch of images optimized by DeepInversion on Yolo-V3 had the same blueish-greenish tinge with strokes as if drawn by a paint brush. I hoped that ssim would increase diversity, but it didn't help too much.

Interestingly, batch similarity is being heavily used by papers for learning backbones with contrastive learning without any labels! e.g https://arxiv.org/abs/2002.05709 uses self similarity between samples in a batch to pull together similar images and pull apart dissimilar images. For generative modelling, you may want to look at GAN papers that might be using self similarity to boost diversity.

from diode.

@animesh-007 Here is the code for SelfSimilarityHook

DIODE/deepinversion_yolo.py

Line 186 in 80a396d

class SelfSimilarityHook():

There are no relevant papers because its just the cosine distance between vectors 😃 source: https://scikit-learn.org/stable/modules/metrics.html#cosine-similarity

If we can represent inputs (e.g images) as vectors (e.g x \in \mathcal{R}^n}) that contain semantically relevant features for those inputs, cosine distance tells us how close they are on the unit circle i.e the angle between the 2 vectors, don't care about the magnitude.

We tried to use it to reduce the similarity between the batch of images that are being optimized (with very little success), so that each image in the batch has different colors and textures. We saw that a batch of images optimized by DeepInversion on Yolo-V3 had the same blueish-greenish tinge with strokes as if drawn by a paint brush. I hoped that ssim would increase diversity, but it didn't help too much.

Interestingly, batch similarity is being heavily used by papers for learning backbones with contrastive learning without any labels! e.g https://arxiv.org/abs/2002.05709 uses self similarity between samples in a batch to pull together similar images and pull apart dissimilar images. For generative modelling, you may want to look at GAN papers that might be using self similarity to boost diversity.

Thanks, @akshaychawla for the detailed answer. I have looked at one GAN paper ref: https://arxiv.org/abs/2105.08584 in the past that uses contrastive learning too.

from diode.

@animesh-007 I am closing the issue since it seems to be resolved. If required, please re-open for further discussion on the same topic, else open a new issue.

from diode.