The downloaded pretrained weights cannot be imported correctly using the jupyter notebook. about clip_prefix_caption HOT 4 CLOSED

Hi @cenjinglun,
It seems that you have a mismatch between the pretrained model and the configuration. Note that these models are using with MLP mapping network and not transformer.

To my opinion, the best way to make sure the pretrained model and configuration is matched, is using the colab notebook we provide.

I'll also will try to make it more clear in the readme.

from clip_prefix_caption.

Thank you very much for pointing out my mistakes.

I want to run it on local, so I downloaded the weights mentioned above. However, because I am not so familiar with the code, I only replaced the model_path in clip_prefix_captioning_inference.ipynb.

model = ClipCaptionPrefix(prefix_length, clip_length=40, prefix_size=640, num_layers=8, mapping_type='transformer')
model_path = 'path/to/coco_weight.pt'       # what I add
model.load_state_dict(torch.load(model_path, map_location=CPU))

If possible, could you tell me the where shoule I load these weights?

from clip_prefix_caption.

You can use the exact code appear in the notebook:
model = ClipCaptionModel(prefix_length) model.load_state_dict(torch.load(model_path, map_location=CPU))
Of course you need you use also other cells

Looking at your code, it seems you used the transformer notebook instead
Please try the clip_prefix_captioning_inference.ipynb notebook

from clip_prefix_caption.

Ok, it works.

Thanks.

from clip_prefix_caption.