Multi-Task Model Fusion via Adaptive Merging

Multi-task model fusion (MTMF) aims to integrate the capabilities of individual models into a unified model. Past approaches either require extensive training and fine-tuning or necessitate that models share the same pre-training and initialization. Recently, several fusion methods have been proposed that do not require extensive training or fine-tuning. These methods can merge multiple independently trained models with different task capabilities into a single multi-task model without increasing the number of parameters. In this work, we identify a common flaw in these fusion methods: they tend to focus on how well the modules of the individual models match before merging while neglecting the representation bias of the merged model. To address this problem, we propose a simple yet effective mitigation method called adaptive merging by representation alignment (AdMbRA). Specifically, we improve the method of weight matching by using representation bias as a constraint and optimize the merging process. Experiments demonstrate that our method can effectively mitigate the representation bias of the merged model, thus improving the performance of each task.