Mergekit support for GPT2 failing about mergekit HOT 2 OPEN

This is somewhat expected. I believe that the issue is that base gpt2 (openai-community/gpt2) doesn't actually have a classification head - it gets randomly initialized when you fine tune it. In this case, though, you actually want to use one of your checkpoints as the base_model anyway - SLERP expects it to be one of the two models you're merging, not a "common ancestor" like other merge methods.

Hope this helps, and sorry for the slow response!

from mergekit.

I have the same issue when merging GPT-2 models.
I can't merge them in either way, using one of my checkpoints as the base model or using a 'common ancestor' as the base model.

It's strange that the GPT-2 model has 'ln_f.bias' and 'ln_f.weight', but they can't be found and lead to runtime errors like the following:
'RuntimeError: Tensor 'ln_f.bias' required but not present in the model.'

When I skipped the 'ln_f' terms, there was another error like the following:
'RuntimeError: Tensor 'h.11.mlp.c_fc.bias' required but not present in the model.'

I believe there might be a naming issue.

I resolved this issue for my usage.
In my loader.index.tensor_paths it was 'transformer.ln_f.bias' instead of 'ln_f.bias'.
I don't know the reason.

from mergekit.