关于Qwen2-72B 全量参数微调所需的显卡下限 about llama-factory HOT 4 OPEN

@hiyouga 后续不知道是否可以参考一下这个repo来改进下长序列微调？https://github.com/jzhang38/EasyContext

from llama-factory.

batchsize 是 1 吗，把 cutoff_len 降低一两倍看看能不能跑起来

from llama-factory.

batchsize 是 1 吗，把 cutoff_len 降低一两倍看看能不能跑起来

是的，per_device_train_batch_size=1，但是我就是需要packing到Qwen支持的最大长度32768呢，是显卡不够吗，贫穷的原因吗，还是说有什么更不占显存的配置呢～

这是我的zero3+offload的配置
{
"bf16": {
"enabled": true
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"weight_decay": "auto",
"torch_adam": true,
"adam_w_mode": true
}
},
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"sub_group_size": 1e9,
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_16bit_weights_on_model_save": true
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"zero_allow_untested_optimizer": true
}

from llama-factory.

目前可能不太支持这么长序列的微调，后续会增加方法。建议先以 8k 长度训练

from llama-factory.