Compatibility with pytorch-transformers for fine-tuning about megatron-lm HOT 6 CLOSED

Hi the config files are the same, however we've had to change some layers' structure and names to account for model parallelism.

We wrote a helper function to allow us to load huggingface/openai weights into our model for debugging purposes. It may work in the reverse direction, but we haven't tested either direction in a while. Please shout if it doesn't work for you.

Note this will not work for model parallel weights, we're still working on our serialization strategies to ship/port model parallel weights.

Raul

from megatron-lm.

Thanks a lot Raul. I will try the function you mentioned and get back on that.

from megatron-lm.

Closing this since it has been a while; I hope that worked out for you! If you have more questions please reopen or start a new issue.

from megatron-lm.

@jaredcasper Any chance on news on the parallel loading part?
I want to import gpt2 pretrained weights and use Megatron to train in parallel

from megatron-lm.

@PyxAI Still in the plans, but not a priority so won't hit the repo for another month or two. Note that we plan to only support loading weights, not optimizer state or anything.

from megatron-lm.

@jaredcasper

I found that the helper function is deleted from the master. I'm also interested in loading huggingface model weights and continue pretraining using Megatron-LM. Do you have current suggestions?

Found the utils.py from the history https://github.com/NVIDIA/Megatron-LM/blob/c882ac61182d423a89d21b453251a20fb7271a67/megatron/utils.py

from megatron-lm.