Model Merging
Model merging creates one checkpoint by mathematically combining parameters or parameter updates from two or more trained models. Common recipes average compatible weights or resolve conflicts among task vectors. The merge operation itself can avoid a new gradient-training run and, unlike an ensemble, normally leaves one model to serve. Useful merging usually assumes compatible architectures, parameter shapes, tokenizers, and often a shared base checkpoint.
Origin and context
Parameter averaging is older than the current LLM wave. Model soups showed in 2022 that averaging multiple fine-tuned models from a shared pre-trained model could improve accuracy and robustness without increasing inference cost. TIES-Merging addressed interference among task-specific updates in 2023. MergeKit then packaged several merging algorithms into an open toolkit in 2024, helping the practice spread through the open-model ecosystem without defining the field by one library.
Why it matters
Merging can consolidate several fine-tuned checkpoints, explore capability trade-offs, or produce a candidate model without the data and compute required for full retraining. It is especially attractive when teams have related variants of the same base model. The result still needs end-to-end evaluation: arithmetic combination does not prove that desired behaviors survive or that unwanted behaviors cancel.
Example
A team has two compatible checkpoints derived from the same base: one tuned for instruction following and another for a domain task. It tests simple averaging and TIES-style merging, then compares each merged checkpoint with both parents on held-out task, safety, calibration, and regression suites. It retains provenance for every input checkpoint and rejects a merge that improves one benchmark while damaging critical behavior elsewhere.
How it differs
Knowledge distillation
Distillation trains a student model on signals from a teacher or teachers. Model merging combines existing parameters directly and need not generate teacher data or optimize a student. A project can use both, but they are different mechanisms with different engineering and validation requirements.
LoRA and QLoRA
LoRA and QLoRA create or train low-rank adapters around a base model. Those adapters or their updates may later be merged, but adapter training is not itself model merging. Compatibility with a shared base remains important.
Evolutionary Model Merging
Evolutionary model merging searches over model combinations or merging recipes with an evolutionary optimization procedure. It is one approach within the broader model-merging field, not an alias for every averaging or task-vector method.
Maturity and evidence
Maturity is rated 3. Multiple peer-reviewed methods and a widely used toolkit establish a durable practice, yet outcomes remain sensitive to checkpoint compatibility, coefficient choices, interference, and evaluation design. There is no universal recipe that predictably composes arbitrary capabilities.
Limits and open questions
Models from different architectures or tokenizers generally cannot be combined by simple weight arithmetic. Even compatible descendants may occupy regions where averaging damages performance, and benchmark gains can hide regressions or contamination. A merge does not prove that desired capabilities will combine cleanly or that unwanted behaviors will disappear. Teams should document the input checkpoints, methods, and coefficients, preserve a reproducible configuration, and evaluate the resulting artifact as a new model rather than describe it as an automatic assembly of skills.
Related terms
References
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeICML / PMLR · 2022-07 · class A
- TIES-Merging: Resolving Interference When Merging ModelsNeurIPS / arXiv · 2023-06-02 · class A
- Arcee's MergeKit: A Toolkit for Merging Large Language ModelsArcee AI / arXiv · 2024-03-20 · class A
- Evolutionary Optimization of Model Merging RecipesNature Machine Intelligence / arXiv · 2024-03-19 · class A
Last updated: 2026-09-03
This term is also covered in the Skills Atlas as model merging skill.