support for larger models about clip.cpp HOT 8 CLOSED

Memory issue will be handled in #8, and I'll have a look at the other crash.

from clip.cpp.

I had an oh! moment just wen pouring coffee into my cup :D Instead of hardcoding a single memory size or running a warmup just to learn the memory requirement, we can decide based on the number of tensors, which essentially indicates the model size. So it sounds like an overengineering, but it's not --it's the only way of automatically setting the correct memory size without any initial delay or lib user's intervention, or the only way I could come up with.

from clip.cpp.

Yeah, but I checked out the fixing branch from my fork until then. We can checkout the upstream master again once it's merged.

from clip.cpp.

Reproduced the issue. Will patch it after testing with other checkpoints of models.

from clip.cpp.

I think this relates to ggerganov/ggml#260

from clip.cpp.

Yes, so let's think of it as a workaround until the proper mecanism is implemented in GGML.

I pushed it in #11

I think we can already merge it.

1. the memory is allocated based on the model variant.
2. the NaN issue is resolved for larger models with the patch size of 14.

from clip.cpp.

nice. yea, it's been an issue ever since i started with llama.cpp 😆

to 2. , you mean one ggerganov/ggml#274 is merged?

from clip.cpp.

#11

from clip.cpp.