Training-free merging of depth-heterogeneous models via activation-driven neuron matching
The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at \href{https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.
Authors
- Dongyang Zhang (ORCID: https://orcid.org/0000-0002-4839-0234)
- Ke Qin (ORCID: https://orcid.org/0000-0001-6174-3877)
- Xin Wu (ORCID: https://orcid.org/0000-0002-0207-0278)
- Shuo An
Institutions
- University of Electronic Science and Technology of China (CN)
Publication Details
- Journal
- International Journal of Machine Learning and Cybernetics
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1007/s13042-026-03382-0
- Primary Topic
- Domain Adaptation and Few-Shot Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Natural Science Foundation of Sichuan Province
- Fundamental Research Funds for the Central Universities