[Bug] Threadgroup memory size exceeds the maximum threadgroup memory allowed about mlc-llm HOT 4 CLOSED

I'm hitting the above error when self.decode_func() is called:

In the provided tokenizer.json, decoder is set to 'null'. We prompt these models with tokens directly and are not doing BPE, so I want some simple passthrough here.

Another clue is that when I initialize DebugChat, I'm greeted with "Warning: Decoder field is not found in tokenizer.json. Use ByteFallback as default."

Any ideas? Could this be a config issue related to tokenizer.json?

from mlc-llm.

It's not a tokenizer issue, but a KVCache kernel issue. I guess that's because the kernel is not fully compatible with fp32

from mlc-llm.

That was it, thank you! Original problem is resolved with different quantization.

The docs are a bit unclear, does the "0" in q0f** mean no weight quantization?

from mlc-llm.

does the "0" in q0f** mean no weight quantization?

@caenopy Yes that's true. Thanks for pointing this out and we'll try to update the docs for better clarity.

from mlc-llm.