Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets
Comparative machine-learning studies often treat the algorithm with the highest observed score as the preferred model, although that choice may depend on the observations available for training. This study examined model-selection stability under limited training samples across six public classification datasets and five algorithms: Logistic Regression, an RBF-kernel SVM, Random Forest, XGBoost, and LightGBM. For each dataset, 30% of the observations were retained as a fixed evaluation set and the remaining 70% formed a development pool. Training fractions of 10%, 20%, 40%, 60%, 80%, and 100% were evaluated in 30 runs per condition. For fractions below 100%, training composition varied across stratified subsets; the 100% condition reused the complete development pool and served as the reference condition. Macro-F1 was the primary model-selection criterion. Dominant winner rate and normalised selection entropy summarized selection stability, while selection regret compared limited-data choices with the full-data reference. Paired Wilcoxon signed-rank tests with Holm correction were used to assess support for the mean-leading algorithm within the experimental protocol. Within-fraction winner agreement was comparatively high in several datasets, but this did not always mean that the same algorithm remained dominant as more training data became available. Landsat changed from Random Forest at 10–20% to LightGBM from 40% onward, whereas Dry Bean changed from Logistic Regression at 10% to SVM-RBF at intermediate fractions and XGBoost at 80–100%. Mean selection regret declined from 0.0097 to 0.0002 for Landsat and from 0.0058 to 0.0008 for Dry Bean between the 10% and 80% conditions. In five limited-data conditions, the mean-leading algorithm was not statistically supported against all competitors after multiplicity correction. The findings show that both training-sample size and composition can change the selected model and the evidence supporting that choice. They describe model-selection stability under training-sample variation conditional on a single fixed evaluation set.
Authors
- Wannaporn Sarapugdi
Institutions
- Rajabhat Maha Sarakham University (TH)
Publication Details
- Journal
- Journal of Computer and Creative Technology
- Published
- 2026-11-15
- DOI
- https://doi.org/10.65205/jcct.2026.e4409
- Primary Topic
- Machine Learning and Data Classification
- Type
- article
- Field-Weighted Citation Impact
- 0.00