CogVLM doesn't seem to load correctly in 4-bit about taggui HOT 17 CLOSED

I use VS Code. I think I'll try PyCharm later.

from taggui.

What OS and GPU are you using? On Linux, CogVLM uses 12.6 GB of VRAM when I load it in 4-bit on my RTX 3090. It was loading correctly on Windows as well, although I did not check the exact memory usage.

from taggui.

OS: win11 23H2
GPU: RTX 4090
Driver: 546.33

from taggui.

That is strange. I just checked on Windows 11 and it's using the expected amount of VRAM (less than 13 GB). It seems as if it is loading the full, unquantized version of CogVLM on your device.

I think the problem may be here

You can check the code from CogVLM. They load the model in 4-bit like this.

I couldn't find any major differences between their code and my code. And it's difficult to troubleshoot the issue because it works correctly on my computer.

Do you have more than one instance of TagGUI running? If you uncheck the Load in 4-bit checkbox on one instance, it will affect the setting across all instances, even if it appears as being checked.

Also, can you verify that the problem persists when you close and restart TagGUI and immediately load CogVLM in 4-bit without loading any other models?

from taggui.

No, I just run one instance of TagGUI.

It's weird.

It seems that "load in 4-bit" doesn't work properly on my computer. I switched to llava-1.5-13b, which also wasn't loading properly and took up about 20GB VRAM.

from taggui.

Unfortunately, I currently cannot figure out what's causing the issue.

You could try manually installing the program by cloning the repository and running pip install -r requirements.txt.

from taggui.

Yep, I have cloned the repo and installing manually.

But I'm not familiar with python.

After I click "Run Auto-Captioner", the program trigger my breakpoint in generate_captions method in class AutoCaptioner, but it doesn't trigger my breakpoint in load_processor_and_model method or run method in class CaptionThread.

from taggui.

It probably has to do with the program creating a new thread to run those functions. Breakpoints at those locations do get triggered for me, though. What editor are you using? I'm using PyCharm.

If you can't get the breakpoints to work, you can try using print() statements.

from taggui.

Well, the breakpoints can be triggered correctly when debugging with PyCharm.

CogVLM is loaded in 4bit, and after loading, the VRAM occupancy is 11.3GB.

However, when the program is executed this code, the VRAM usage starts to climb, and it has climbed to ~30GB. I don't understand what happened.

from taggui.

That's interesting. Could you step into the generate() function to check which part of it causes the VRAM usage to increase?

from taggui.

Are you using any nondefault generation parameters (like number of beams > 1)?

from taggui.

I use the parameters like their Demo.

top_p: 0.8
top_k: 5
temperature: 0.9
"use sampling" is checked
max tokens: 225

Is it a parameter setting problem?

from taggui.

Well, I think it may be caused by parameters.

When I use beam search and set "Number of beams" to 1, it only takes up 13.1 GB VRAM.

from taggui.

I don't think those parameters should cause any problems. Just make sure the number of beams is 1 for now.

You could try disabling sampling and see if that fixes the issue. It's probably not related, though.

from taggui.

When I use beam search and set "Number of beams" to 1, it only takes up 13.1 GB VRAM.

Were you using more beams before?

from taggui.

I set "Number of beams" to 6 before and "use sampling" is checked before.

I do some test and it seems that whether "use sampling" is checked or not will not have much impact on the VRAM usage, but increasing the "Number of beams" will significantly increase the VRAM usage.

When "Number of beams" is set to 4, ~23GB of VRAM is occupied.
When "Number of beams" is set to 6, ~31GB of VRAM is occupied.

from taggui.

Yes, increasing the number of beams can have that effect. It's a tradeoff between accuracy and speed/memory usage. I had completely forgotten about that.

I guess that solves the issue then. Thanks for being patient and figuring it out with me!

from taggui.