Rethinking data selection strategies for more accurate software effort prediction using the ISBSG dataset

A common practice in studies on software development effort prediction involving the ISBSG dataset is upfront data selection by keeping only high-quality cases according to the Data Quality Rating and UFP Rating and the narrow group of predictor attributes having no or very few missing values. Hence, a substantial part of the dataset is discarded. This study investigates the impact of training data quality on the performance of models for software effort prediction. It explores whether less restrictive data selection improves predictive accuracy. Model performance was evaluated with a standardised accuracy, a “win–tie–loss” approach, and a matched-pairs rank biserial correlation coefficient. The non-parametric Scott-Knott effect size difference test provided rankings of data selection strategies and prediction techniques. Using a larger training subset, i.e., including more cases despite a small fraction with low ratings and more predictor attributes despite some of them even with up to 80% of missing values, not only did not degrade model performance but, on the contrary, in many cases, improved it. For most models, the larger size of the training dataset was more beneficial than the smaller one, with only high-quality data. Hence, if predictive accuracy is the priority, training models on all cases (even with low ratings) and a broad set of attributes (up to 70%–80% of missing values) is justified. The best-performing prediction techniques were SVM, XGBoost, and neural networks, depending on the data selection strategy. The provided rankings offer helpful guidance for setting up future effort prediction studies. • Selecting only high-quality data reduces the training subset of the ISBSG repository. • Less restrictive training data selection improves effort prediction accuracy. • SVM, XGBoost, and neural networks performed the best, depending on the data variant. • Obtained results were stable across data splits, models, and data variants. • Provided rankings of data variants and models help set up future studies.