there are an plan to release the training part? about glm-130b HOT 4 CLOSED

Thanks for your attention! We are still in active development for quantization and other explorations. The training code will be released after we are done, maybe later this month.

from glm-130b.

Unlike OPT-175B and BLOOM-176B, we use 40G A100 instead of its 80G version, so the GPU memory could be very tight for training. We have tried numerous combinations of parallelism / hidden size / layers and ended up using 4-way tensor and 8-way pipeline parallelism for the best training throughput, approximately 135 TFLOPS / GPU. Adam optimizer with ZeRO stage 1 is used in actual training. At least 40 nodes are required to start training. Using optimizers like Adafactor or 8-bit Adam could further reduce the minimum requirement to 24 nodes, however, we did not use them for best convergence.

from glm-130b.

@Sengxian 1. Is it convenient to introduce the basic training settings? For example, the number of model parallel mp and zero stage in training, the minimum number of nodes required for training?

from glm-130b.

@Sengxian hi, has this plan delayed?

maybe later this month.

from glm-130b.