How to use the code for multinode inference? about glm-130b HOT 4 CLOSED

See https://github.com/THUDM/GLM-130B#model-weights for checkpoint conversion script. You may use INT4 quantization for 3*40G A100s.

from glm-130b.

Hi thanks for the response. But I guess I didn't make myself clear.

I have enough gpus so no quantization is needed for me.

However, those gpus exist among multiple nodes (machines) managed under a single slurm manager.

So what I need is multi-node inference. The model weights need to exist among the gpus in multiple nodes.

The code in this repo assumes single node: no_procs_per_node = mp_size

Do you have advice on adapting the code to multinode inference?

from glm-130b.

You can adapt this multi-node evaluation script to a multi-node generation. With arguments like NUM_WORKERS=8 and NUM_GPUS_PER_WORKER=1.

from glm-130b.

much thanks. but I did it with just torchrun with some rdzv arguments

from glm-130b.