Broken docker container image about llm-guard HOT 8 CLOSED

Sorry, didn't have time yet to check. My instance has 60GB of disk but only 2 of RAM - that might be the problem.

from llm-guard.

Hey @baggiponte , thanks for reaching out.

Can you please share more information about your environment? Is it Macbook M1 or something else? Does it break immediately after starting?

It seems that Docker doesn't have enough storage to load models. I tried to run this command on my machine and had no errors.

from llm-guard.

Oh lol that might be it. Will prune everything and come back to it :)

from llm-guard.

heya, I pruned the whole docker partition and ran again the command but it failed. I am running on a M3 Mac.

from llm-guard.

Is it the same error? It needs quite a lot of space, can you check also the docker config if it has enough storage allocated?

from llm-guard.

We found another problem in our docker image which was fixed yesterday: e12b9a3

from llm-guard.

Hey there! Sorry for popping up late. The breakdown is definitely due to the container having allocated too little RAM. I bumped mine from 2GB to 8GB and the download proceeds well beyond the point it used to (but stil crashes).

Perhaps providing an approximate memory usage could be of help. Both in the docs for the container image but also for the scanner - something like an attribute or a staticmethod that provides one.

Feel free to close this issue and/or move it to another one or a discussion.

from llm-guard.

Hey @baggiponte , will add that information to the doc. And actually we also realized the image is quite heavy, and we will optimize it with a few tricks.

from llm-guard.