Investigating the Effect of Class Imbalance on the Performance of Self-Admitted Technical Debt Prediction Models

Self-Admitted Technical Debt (SATD) refers to situations where developers explicitly acknowledge design or implementation compromises in source code comments. Although SATD detection has been extensively studied in recent years, most datasets are severely imbalanced, leading models to bias toward the majority class. This paper empirically investigates how class imbalance affects the performance and stability of SATD prediction models. We design a dataset generation algorithm to construct datasets with varying imbalance ratios and benchmark four representative approaches: unsupervised pattern matching (MAT), traditional text mining (TM), maximum-entropy-based natural language processing (NLP), and convolutional neural networks (CNN). We evaluate these approaches under both many-to-one (MTO) and one-to-one (OTO) prediction scenarios. Experimental results show that CNN achieves competitive performance and stability in cross-project (MTO) prediction, while MAT remains stable in single-project (OTO) settings and under highly imbalanced conditions. TM and NLP methods can reach local peaks under moderate-to-high imbalance ratios. These findings delineate the trade-offs among methods, offering actionable guidance for selecting SATD detection models under varying imbalance conditions.