Conceptual Captions Training about clip_prefix_caption HOT 9 OPEN

Hi @goel-shashank,
Are you using our default parameters?

Did you try both the GPT-2 fine-tuning and the frozen GPT-2?

from clip_prefix_caption.

Hi @rmokady,
I tried the default parameters. Do you have the training logs for your run? One thing I'm certainly doing differently is that I have trained a separate CLIP model (RN50 with 20% imagenet zero-shot accuracy) which is trained on CC3M (not OpenAI's pretrained). The prefixes are generated from this model. I don't think this should be causing these issues.

from clip_prefix_caption.

For COCO where we train both prefix and GPT-2 the loss got to 1.47
Unfortunately, the logs for the Conceptual got left on an old server and cannot access these anymore
5 epochs for 3M images is a lot using the standard clip

Anyway, outputting the same sentence for any prefix usually means there is a bug somewhere

from clip_prefix_caption.

As I mentioned, I was able to fit on a batch of 512 image-caption pairs and everything works out so I don't think there is any logical issue with the pipeline. But still, I will check everything for once. Closing this issue! Please let me know if you find something useful!

from clip_prefix_caption.

Hi @goel-shashank,
I find some logs for conceptual captions. This is with the resnet CLIP:

from clip_prefix_caption.

This is with the Vit CLIP:

from clip_prefix_caption.

I have the same problem with my own dataset. It keeps generating similar captions...

from clip_prefix_caption.

Hi, I have the same problem for Conceptual Captions + frozen model. Do you have loss values for that scenario? All the inputs end up converging to the same prefix. Thanks!

I followed the README and ran:

python parse_conceptual.py --clip_model_type ViT-B/32 --data_root /path/to/conceptual_captions --num_threads 100

and then

python train.py --only_prefix --data /path/to/conceptual_captions/conceptual_clip_ViT-B_32_train.pkl --out_dir /path/to/output_dir --mapping_type transformer --num_layers 8 --prefix_length 40 --prefix_length_clip 40

from clip_prefix_caption.

@surisdi Did you manage to reproduce the results?

from clip_prefix_caption.