Can this use convNeXt architecture? about clip.cpp HOT 4 CLOSED

I see no reasons it doesn't do well in the CV world,

Sure, what I meant was convolution operations in GGML currently have fixed kernel sizes / strides. We need to figure out a way of implementing generic convolutions, then implementing any convolution-based model should be straightforward. Luckily, ViT models usually use a single type of convolution operation and the rest is plain transformer blocks. But traditional models such as ResNet have a very rich set of convolution ops.

Do you have any local test data avaialble on the inference speed comparison between torch&CUDA vs clip.cpp

I implemented a benchmarking utility to measure the zero-shot labeling performance with acc@1 and acc@5 metrics as well as inference time, but haven't done a full comparison with Pytorch --I'm trying to decide on a good test set. I'm also currently working on batch inference support, maybe it would be more appropriate to include single-instance and batch inference performances in benchmarking.

from clip.cpp.

Unfortunately not yet. Currently this implements only ViT architecture. GGML's support for convolutions is currently limited, and as I implement more models I'd like to contribute back to GGML. I have a few architectures in mind to implement after CLIP, but community demand is also important.

from clip.cpp.

Thanks.
I think ggml is proven to be very efficient in the LLM inference, I see no reasons it doesn't do well in the CV world, plus the quantization is quite handy. k-quant is even better yet that's specific to transformers.
Do you have any local test data avaialble on the inference speed comparison between torch&CUDA vs clip.cpp &cuBLAS on the same ViT architecture?

from clip.cpp.

Unfortunately not yet. Currently this implements only ViT architecture. GGML's support for convolutions is currently limited, and as I implement more models I'd like to contribute back to GGML. I have a few architectures in mind to implement after CLIP, but community demand is also important.

so,How's it going now?How's it going now?I came here from stable diffusion.cpp, ggml_conv_2d caused sd to not run on gpu.Its optimization needs some inspiration

from clip.cpp.