Session crashed on colab about mixtral-offloading HOT 4 CLOSED

Hey! Just tried running the notebook (in the offload_per_layer = 5 setting) and everything works for me. Have you tinkered with the original notebook in any way? If not, try restarting the session and running it again starting with the model initialization cell.

from mixtral-offloading.

Hey!

Have you managed to solve this issue? If no, can you please provide some more information:

1. On which cell exactly did the notebook crush with offload_per_layer = 4?
2. How many of RAM and GPU VRAM did you have in Google Colab?

Note that our demo notebook should run normally in Google Colab with offload_per_layer = 4, but will crush sometimes with offload_per_layer = 5. The latter option is made for local run with low VRAM.

from mixtral-offloading.

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

Hqq_aten is not required as we have custom triton kernels for GEMV.

from mixtral-offloading.

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

from mixtral-offloading.