Fine-Tuning Code Language Models to Detect Cross-Language Bugs

With the rapid development of software engineering technologies, multilingual programming, i.e., developing a program using multiple programming languages (PLs) collaboratively, has become increasingly popular. This practice is often adopted to reuse existing code, to leverage advantages of specific PLs, to satisfy various software quality needs, and to enhance software development efficiency. However, it also introduces cross-language bugs (CLBs), bugs caused by interactions between multiple PLs. Most existing bug detection tools and techniques are primarily designed for a single PL and are less effective at detecting CLBs. Given the significant advancements of Pre-trained Language Models (PLMs) in the code domain, this work investigates the performance of pre-trained Code Language Models (CodeLMs) in CLB prediction tasks. To this end, we designed and developed a cross-language code identification tool (CLCFinder), and used it to collect and construct a CLB dataset, which involves combinations of three PLs (i.e., Python-C/C++, Java-C/C++, and Python-Java) with nine types of interaction mechanisms. We fine-tuned 13 popular CodeLMs in different sizes on this dataset, in which 80% of the data was used for fine-tuning, 10% for validation, and 10% for testing. We then evaluated their performance in CLB detection. Additionally, we compared the fine-tuning performance of five top-performing models on current state-of-the-art (SOTA) single-language bug datasets to assess their application in CLB detection. Furthermore, we evaluated factors affecting the performance of fine-tuned CodeLMs in CLB detection, including the size of the fine-tuning dataset and the length of token sequence. Finally, we explored the impact of code comments on the performance of fine-tuned CodeLMs. The experimental results are that: First, all 13 CodeLMs exhibited varying degrees of performance improvement after fine-tuning. In particular, after fine-tuning, the UniXcoder-base model achieved the best performance, with an F1 score of 0.7407. Notably, within the scope of our experimental setup, small CodeLMs tended to achieve better performance than large CodeLMs, while large CodeLMs showed limited improvement. Second, the fine-tuned CodeLMs on single-language bug datasets performed poorly in CLB detection, further demonstrating the significant feature differences between CLBs and single-language bugs. Third, enlarging the fine-tuning dataset size significantly enhanced the model performance in CLB detection, but increasing token sequence length did not necessarily improve the model performance. Finally, regarding the impact of code comments, the performance varied significantly across different CodeLMs. Specifically, some CodeLMs’ performance was improved when fine-tuned with code comments, while others showed degraded performance.