Merging task-specific adapters in code LLMs for automated program repair

Large Language Models (LLMs) have shown good performance in several software development related tasks such as program repair, documentation, code refactoring, debugging, and testing. Adapters are specialized, small modules designed for parameter efficient fine-tuning of LLMs for specific tasks, domains, or applications without requiring extensive retraining of the entire model. These adapters offer a more efficient way to customize LLMs for particular needs, leveraging the pre existing capabilities of the large model. Merging LLMs and adapters has shown promising results for various natural language domains and tasks, enabling the use of the learned models and adapters without additional training for a new task. This research proposes continual merging and empirically studies the capabilities of merged adapters in Code LLMs, specially for the Automated Program Repair (APR) task. The goal is to gain insights into whether and how merging task-specific adapters can affect the performance of APR. In our framework, MergeRepair, we merge multiple task-specific adapters using three different merging methods and evaluate the performance of the merged adapter for the APR task. Particularly, we employ two main merging scenarios for all three techniques, (i) merging using equal-weight averaging applied on parameters of different adapters, where all adapters are of equal importance; and (ii) our proposed approach, continual merging, in which we sequentially merge the task-specific adapters and the order and weight of merged adapters matter. By exploratory study of merging techniques, we investigate the improvement and generalizability of merged adapters for APR. Through continual merging, we explore the capability of merged adapters and the effect of task order, as it occurs in real-world software projects. The results show that we can gain performance improvement on APR benchmark by merging APR adapter with other similar task adapters. Additionally, we can observe the effectiveness of the continual merging approach if the order of the merged tasks is selected carefully.