CutMix for Image Captioning about cutmix-pytorch HOT 4 CLOSED

@vinson2233
Hi, thank you for having interest in our work!

Hi, I really like your paper. But I have a question.
I understand how CutMix is applied to the image classification task, but I'm confused about how do you implement CutMix for Image Captioning task
Given λ that sampled from the uniform distribution (0, 1), how do you merge 2 captions?

It will be great if you could provide an example like :
with λ = 0.2, y1 = "A man wearing a blue jeans standing inside the bus", y2 = "A bird is flying in the sky" will be combined into xxx
Thanks :)

Actually, our Image Captioning experiments do not apply CutMix during training time. Only the pretrained model trained with CutMix is used to validate its transferrable ability.
I think your idea to combine two caption labels is quite interesting, and hope to see your further work!

from cutmix-pytorch.

From my understanding is, NIC workflow is Image -> CNN - > LSTM -> Caption
So the CNN part replaced with your CutMix Pretrained model ?

Yes, thanks.

from cutmix-pytorch.

Thanks for the fast response :)
So this section of the paper,

Transferring to MS-COCO image captioning:
We used Neural Image Caption (NIC) [43] as the base model for image captioning experiments. We have changed the backbone
network of encoder from GoogLeNet [43] to ResNet-50.

From my understanding is, NIC workflow is Image -> CNN - > LSTM -> Caption
So the CNN part replaced with your CutMix Pretrained model ?

from cutmix-pytorch.

Great, thank you :)

from cutmix-pytorch.