Which CLIP model? about clip_prefix_caption HOT 5 CLOSED

Hi @YovaKem,
We used both, and the results were similar.

But if anyone observed differently I would be interested to hear.

from clip_prefix_caption.

Thanks, @rmokady! If I understand correctly the model available at this address: 14pXWwB4Zm82rsDdvbGguLfx9F8aM7ovT as used here uses ViT-B/32 as the image encoder for Conceptual Captions data. Is there an address to download the ResNet50x4 model for Conceptual Captions? Have you made that available/are you able to share it with me? Thanks!

from clip_prefix_caption.

notebooks/clip_prefix_captioning_inference.ipynb contains the ResNet50x4 model , while also train the GPT-2 and employ MLP mapping network.

from clip_prefix_caption.

Ack, sorry, I'm not able to see where, can you point me to the exact location?

These are the only lines I can see where a model is being downloaded:

if pretrained_model == 'Conceptual captions':
  downloader.download_file("14pXWwB4Zm82rsDdvbGguLfx9F8aM7ovT", model_path)
else:
  downloader.download_file("1IdaBtMSvtyzF0ByVaBHtvM0JYSXRExRX", model_path)

Here, it's not clear whether the models are trained on ViT or ResNet, but then two cells down a ViT encoder is used to encode the images regardless of which model was loaded

clip_model, preprocess = clip.load("ViT-B/32", device=device, jit=False)

So I'm assuming both models linked above use the ViT encoder.

What would that alpha-numerical string be for downloading a ClipCaps model (i.e. the parameters of the MLP and finetuned GPT2) based on ResNet50x4? Thanks a lot for your help!

from clip_prefix_caption.

Hi @YovaKem,
Indeed I was wrong and pointed to wrong files

We didn't publish Conceptual Captions model based on ResNet and we currently not plan to as there is no improvement
Sorry for the confusion

from clip_prefix_caption.