Requirements for Model Merging

[eunbin079]

What common characteristics should models have to be eligible for task arithmetic merging?
Besides having the same architecture in the model config.json, what other conditions should be met? Can you provide a guide?

[cg123]

The models should have the same architecture and the same parameter sizes. A better way to think of it, though, is that to merge models they all need to have been fine tuned from some common "ancestor" model. For example, any set of models that can be traced back to Mistral-v0.2-7B (through however long a chain of fine tuning and merging) should be compatible. On the other hand, a model fine tuned from a Yi base and one from a Llama base will never be compatible even if the architecture and parameter sizes are exactly the same.

Hope this helps clear things up a bit.

[eunbin079]

thank for replying !!