The Impact of Class Noise-handling on the Effectiveness of Machine Learning-based Methods for Build Outcome and Code Change Request Predictions

Machine learning-based methods for optimizing build processes and code review processes are commonly used to accelerate feature delivery to end-users in modern software engineering. These methods leverage large volumes of historical code changes to train models on predicting and preventing issues in the code-base that could delay code integrations. The objective of this paper is to improve these methods by reducing the impact of noise on their predictive performance. In this paper, we investigate the impact of several class noise-handling techniques on improving the predictive capabilities of machine learning models for build outcome and negative code review prediction tasks. We conduct a series of computational experiments using data from 110 Java open-source projects and examine the effectiveness of two removal-based statistical techniques – Majority Filter (MF) and Consensus Filter (CF) – and two corrective techniques – domain knowledge-based (DB) and CleanLab. Our results show that removal-based techniques significantly improve model predictive performance for both build outcome prediction and negative code review predictions. For build outcome prediction, applying the MF improved the F1-score from 82% to 97%, and MCC from 0.13 to 0.58. In negative code review predictions, MF improved the F1-score from 17% to 53%, and MCC from -0.03 to 0.57. DB proved effective primarily in the context of code review comments, but less so for build outcome predictions. CleanLab demonstrated a more conservative approach to noise detection, retaining complex code constructs that other techniques identified as noisy. While applying CleanLab resulted in more consistent predictions, its overall impact on model predictive performance was more moderate compared to removal-based techniques. Additionally, our findings show that hyperparameter tuning, when applied independently or in combination with CleanLab can improve model performance. However, these improvements did not surpass the gains achieved through applying the removal-based techniques alone. We conclude that applying removal-based techniques to the training data of code changes is necessary to leverage the prediction of build outcomes and negative code review comments.