Support image-only about clip.cpp HOT 8 OPEN

zsl (zero-shot labeling) examples does that. We need to pass candidate class names with --text argument and zsl is scoring those classes.

from clip.cpp.

Oh got it. This is called image captioning. There exist numerous models for it, but state-of-the-art results come from models like LLaVA. It is basically CLIP + LLaMA bridged with a linear layer. I have an idea like creating another project to combine clip.cpp with llama.cpp to achieve efficient inference of LLaVA, but this might be delayed for a week or so because there are some other features I'd like to implement in this repo before that.

from clip.cpp.

If you don't have candidates, CLIP model won't work for image classification. Then I guess your best bet would be to hardcode class names from a common dataset such as OpenImages by modifying zsl example.

p.s.: I'm planning to experiment with a transfer learning method on the edge to train a single head layer on top of the CLIP backbone for image classification, but I don't know when yet.

from clip.cpp.

sounds like you want to feed the image embedding to a llm. (like llava)

from clip.cpp.

I do not have a general list of candidates to provide

from clip.cpp.

Can you go into more detail?

common dataset such as OpenImages

Like go to OpenImages and generate a text prompt with all its classes in to the command line prompt?

from clip.cpp.

Yes, or only the classes that you are actually expecting to appear in the image. This is how zero-shot labeling is supposed to work.

If you could describe your exact use case, I'd try to make a more detailed comment.

from clip.cpp.

Here’s an example. I want to create an discord bot that watches for a reaction on the image and posts a message explaining what the image looks like for people who have trouble seeing, but know what the words mean.

from clip.cpp.