It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-Objective Optimization

In this paper, we introduce a novel approach for addressing the multi-objective optimization problem in large language model merging via black-box multi-objective optimization algorithms. The goal of model merging is to combine multiple models, each excelling in different tasks, into a single model that outperforms any of the individual source models. However, the effectiveness of conventional model merging methods is constrained by human intuition or domain knowledge. While existing optimization-based model merging methods can automatically search for model merging parameter configurations, they often struggle to find a satisfactory configuration within a limited evaluation budget. To address this challenge, we propose a novel and sample-efficient automated model merging method, named MM-MO. This method leverages multi-objective Bayesian optimization algorithms to autonomously search for great merging configurations across various tasks. In MMMO, we proposed an enhanced acquisition strategy and an auxiliary optimization objective to improve the search process. Our enhanced acquisition strategy integrates a weak-to-strong method to refine the acquisition function, enabling previously evaluated superior configurations to guide the search for new ones. Meanwhile, Fisher information is utilized to further filter these configurations, increasing the possibility of finding high-quality merging configurations. Additionally, we design a sparsity metric as an auxiliary optimization objective, further enhance the models generalization performance across different tasks. We conducted comprehensive experiments with other mainstream model merging methods, demonstrating that the proposed MMMO algorithm is competitive and effective in achieving high-quality model merging.