Security Bug Report Prediction Within and Across Projects: A Comparative Study of BERT and Random Forest

Early detection of security bug reports (SBRs) is crucial for preventing vulnerabilities and ensuring system reliability. While machine learning models have been developed for SBR prediction, their predictive performance still has room for improvement.In this study, we conduct a comprehensive comparison between BERT and Random Forest (RF), a competitive baseline for predicting SBRs. The results show that the highest G-measures are 70% for RF and 66% for BERT. In individual projects, RF outperforms BERT with a 34% higher G-measure. Adding only SBRs from various projects improves both models. However, including both security and nonsecurity bug reports boosts BERT to its best performance, surpassing RF, while RF's performance drops significantly to 46%. Notably, BERT achieves a 62% G-measure for "cross-project" SBR prediction using reports from multiple projects, which is substantially higher than RF.