What is the function of sample captions and captions per image about a-pytorch-tutorial-to-image-captioning HOT 6 CLOSED

Hi, consider that our dataset contains n_i images, and each image has a bunch of captions. The number of captions available for each image may vary.

captions_per_image is a number you use so that:

• you can restrict the size of the dataset by limiting the number of captions to use per image. So if you want to use all images for training, but you want a smaller dataset, you can use only (say) 3 captions per image.
• all images have the same number of captions.

The function in utils.py samples the captions in the following way:

• If an image has fewer than captions_per_image captions available, increase the number by sampling with replacement, similar to how it's done in NeuralTalk2. (Note that I can use random.choices() to do this directly instead of using random.choice() in a for loop. I just wasn't aware of this function.)
• Otherwise, simply choose captions_per_image captions without replacement, using random.sample().

So, if we have n_i images, and captions_per_image captions for each image, the total number of captions in the final data is n_c = n_i * captions_per_image. Since the images and captions are saved in the same order, the ith caption will correspond to the i // captions_per_imageth image.

from a-pytorch-tutorial-to-image-captioning.

In a few days, I'll add/update comments in the relevant portions of the code so this is clearer.

from a-pytorch-tutorial-to-image-captioning.

The resulting files do contain the sampled captions, but they are not grouped in 5s - they are flattened lists.

So, if you have 100 images, and you sample 5 captions per image, there are a total of 500 captions, right?

Then,

TRAIN_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json contains the true word lengths of the 500 captions.

TRAIN_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json contains the 500 captions, where each caption is a list of encoded numbers.

So both these lists have 500 elements. The first 5 elements in both files correspond to Image 1, the second 5 elements correspond to Image 2, etc.

The images are stored in the HDF5 file, and there are a 100 of them. They're in the same order as the captions. For example, the 143rd element from the first two files will correspond to the 143 // 5 = 28th image from this file.

btw nice jakiro avatar I also play dota

Thanks! It's great to hear you play too. He's my favorite hero.

from a-pytorch-tutorial-to-image-captioning.

after discussing and reading your explanation
I open the karpathy json files and beautify the json
I just realize one image could have 5 captions

my mistake is misunderstanding about the word "captions" (english is not my native tongue)
I assume the "captions" is the "token"
so when there is a image with this captions A boy is sitting on a brightly colored chair next to a child 's book playing a guitar

I thought the captions per image will limit it to A boy is sitting on

karpathy _json_file.json

{
    "sentids": [14420, 14421, 14422, 14423, 14424],
    "imgid": 2884,
    "sentences": [{
      "tokens": ["a", "boy", "is", "sitting", "on", "a", "brightly", "colored", "chair", "next", "to", "a", "child", "s", "book", "playing", "a", "guitar"],
      "raw": "A boy is sitting on a brightly colored chair next to a child 's book playing a guitar .",
      "imgid": 2884,
      "sentid": 14420
    }, {
      "tokens": ["a", "toddler", "is", "sitting", "on", "a", "red", "checked", "chair", "whilst", "playing", "a", "guitar"],
      "raw": "A toddler is sitting on a red checked chair whilst playing a guitar .",
      "imgid": 2884,
      "sentid": 14421
    }, {
      "tokens": ["a", "young", "child", "is", "sitting", "in", "a", "colorful", "chair", "with", "a", "guitar", "in", "his", "hands", "and", "a", "book", "sitting", "next", "to", "him"],
      "raw": "A young child is sitting in a colorful chair with a guitar in his hands and a book sitting next to him .",
      "imgid": 2884,
      "sentid": 14422
    }, {
      "tokens": ["a", "young", "child", "playing", "guitar", "on", "the", "chair"],
      "raw": "A young child playing guitar on the chair",
      "imgid": 2884,
      "sentid": 14423
    }, {
      "tokens": ["young", "barefoot", "boy", "playing", "a", "guitar", "setting", "in", "a", "large", "muti", "colored", "chair", "with", "a", "picture", "of", "three", "pigs", "beside", "him"],
      "raw": "Young barefoot boy playing a guitar setting in a large muti colored chair with a picture of three pigs beside him .",
      "imgid": 2884,
      "sentid": 14424
    }],
    "split": "train",
    "filename": "2201222219_8d656b0633.jpg"
  }

thank you so much @sgrvinod for your explanation, quick reply and this awesome tutorial

from a-pytorch-tutorial-to-image-captioning.

thanks for the explanation of the code now I understand it, but I still confused though

if captions_per_image is limiting the caption that we use per image and utils.py is sampling the caption

why the final result is still the real caption(not limited to 5 and not randomized/sampled) ?

create_input_files.py

 create_input_files(dataset='coco',
                              karpathy_json_path='../caption data/dataset_coco.json',
                              image_folder='/media/ssd/caption data/',
                              captions_per_image=5,
                              min_word_freq=5,
                              output_folder='/media/ssd/caption data/',
                              max_len=50)

I open the TRAIN_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json

the number is like this

[13, 13, 14, 10, 12, 12, 12, 13, 11, 11, 15, 11, 11, 11, 11, 14, 23, 10, 11, 11, 11, 13, 10, 11, 16, 16, 12, 13, 11, 11, 11, 12, 12, 10, 13, 15, 10, 10, 13, 13, 12, 11, 14, 13, 13, 11, 10, 10, 14, 14, 12, 10, 15, 10, 11, 12, 15, 11, 22, 12, 10, 14, 11, 13, 12, 10, 13, 14, 12, 11, 10, 10, 11, 11, 11, 13, 15, 12, 11, 10, 14, 13, 16, 14, 14, 13, 13, 10, 10, 11, 10, 27, 12, 16, 13, 10, 14, 10, 11, 13, 10, 10, 12, 11, 11, 12, 11, 12, 12, 12, 15, 19, 11, 13, 12, 18, 12, 13, 13, 12, 13, 15, 16, 12, 11, 13, 14, 12, 12, 14, 10, 10, 12, 13, 10, 12, 10, 10, 11, 11, 12, 10, 11, 11, 13, 15, 12, 10, 11, 14, 12, 10, 11, 20, 10, 11, 12, 16, 11, 10, 12, 17, 19, 13, 15, 11, 13, 13, 13, 12, 10, 13, 15, 11, 14, 11, 13, 13, 11, 14, 13, ...]

so is the captions file TRAIN_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json

[[9488, 1, 38, 39, 1, 40, 6, 41, 42, 43, 1, 44, 9489, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
[9488, 1, 38, 39, 1, 48, 40, 43, 1, 45, 47, 44, 9489, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
[9488, 1, 38, 51, 1, 44, 3, 14, 16, 17, 1, 52, 53, 9489, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[9488, 1, 38, 43, 1, 45, 46, 47, 44, 9489, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ....]

btw nice jakiro avatar I also play dota

from a-pytorch-tutorial-to-image-captioning.

Glad you were able to figure it out, and thanks!

from a-pytorch-tutorial-to-image-captioning.