Training-free merging of depth-heterogeneous models via activation-driven neuron matching

The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at \href{https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.

Authors

Institutions

Publication Details

Journal
International Journal of Machine Learning and Cybernetics
Published
2026-11-18
DOI
https://doi.org/10.1007/s13042-026-03382-0
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Training-free merging of depth-heterogeneous models via activation-driven neuron matching

Dongyang Zhang, Ke Qin, Xin Wu, Shuo An
International Journal of Machine Learning and Cybernetics
Domain Adaptation and Few-Shot Learning
article

Training-free merging of depth-heterogeneous models via activation-driven neuron matching

Dongyang Zhang, Ke Qin, Xin Wu, Shuo An
article en

Abstract

The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at \href{https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.

International Journal of Machine Learning and CyberneticsVol. 17(12)
University of Electronic Science and Technology of China (CN)
Natural Science Foundation of Sichuan Province, Fundamental Research Funds for the Central Universities
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon