RL–Fusion: The Large Language Model Fusion Method Based on Reinforcement Learning for Task Enhancing

Model fusion is a technique of growing interest in the field of machine learning, which constructs a generalized model by merging the parameters of multiple independent models with different capabilities without the need to access the original training data or perform costly computations. However, during model fusion, when the number of parameters in a large language model is high, the dimension of the parameter space increases, which makes it more challenging to find the optimal combination of weights. Meanwhile, there is considerable potential for further development in sustainable optimization schemes for task-specific performance enhancement through model fusion in this area. In this paper, we propose a large-scale language model fusion approach based on task-enhanced reinforcement learning (RL–Fusion) to efficiently explore and optimize model fusion configurations. The key innovation of RL–Fusion lies in its use of reinforcement learning to guide parameter selection during model fusion, enabling a more intelligent and adaptive exploration of the parameter space. Additionally, RL–Fusion introduces a dynamic evaluation mechanism that adjusts the evaluation dataset in real-time based on feedback from SOTA models, ensuring continuous enhancement of domain-specific capabilities. RL–Fusion outperforms the baseline model by improving 1.75% in the MMLU benchmark test, 1.8% in the C-eval test, and 1.8% in the Chinese Named Entity Recognition (NER) test on the Yayi NER dataset by 16%. The results show that RL–Fusion is an effective and scalable model fusion solution that improves performance without increasing the computational cost of traditional optimization methods and has a wide range of applications in AI research and practice.