Beginner Question: Merge 2 7B models to make 13B model about mergekit HOT 2 CLOSED

Yes, it is possible - people have had some pretty cool successes with this sort of model stacking lately.

For example, here's the config Undi95 used to make Mistral-11B-OpenOrcaPlatypus (one of the steps in his Mistral-11B-OmniMix):

slices:
  - sources:
    - model: Open-Orca/Mistral-7B-OpenOrca
      layer_range: [0, 24]
  - sources:
    - model: akjindal53244/Mistral-7B-v0.1-Open-Platypus
      layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16

from mergekit.

Great, thank you!

from mergekit.