Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets

Comparative machine-learning studies often treat the algorithm with the highest observed score as the preferred model, although that choice may depend on the observations available for training. This study examined model-selection stability under limited training samples across six public classification datasets and five algorithms: Logistic Regression, an RBF-kernel SVM, Random Forest, XGBoost, and LightGBM. For each dataset, 30% of the observations were retained as a fixed evaluation set and the remaining 70% formed a development pool. Training fractions of 10%, 20%, 40%, 60%, 80%, and 100% were evaluated in 30 runs per condition. For fractions below 100%, training composition varied across stratified subsets; the 100% condition reused the complete development pool and served as the reference condition. Macro-F1 was the primary model-selection criterion. Dominant winner rate and normalised selection entropy summarized selection stability, while selection regret compared limited-data choices with the full-data reference. Paired Wilcoxon signed-rank tests with Holm correction were used to assess support for the mean-leading algorithm within the experimental protocol. Within-fraction winner agreement was comparatively high in several datasets, but this did not always mean that the same algorithm remained dominant as more training data became available. Landsat changed from Random Forest at 10–20% to LightGBM from 40% onward, whereas Dry Bean changed from Logistic Regression at 10% to SVM-RBF at intermediate fractions and XGBoost at 80–100%. Mean selection regret declined from 0.0097 to 0.0002 for Landsat and from 0.0058 to 0.0008 for Dry Bean between the 10% and 80% conditions. In five limited-data conditions, the mean-leading algorithm was not statistically supported against all competitors after multiplicity correction. The findings show that both training-sample size and composition can change the selected model and the evidence supporting that choice. They describe model-selection stability under training-sample variation conditional on a single fixed evaluation set.

Authors

Institutions

Publication Details

Journal
Journal of Computer and Creative Technology
Published
2026-11-15
DOI
https://doi.org/10.65205/jcct.2026.e4409
Primary Topic
Machine Learning and Data Classification
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets

Wannaporn Sarapugdi
Journal of Computer and Creative Technology
Machine Learning and Data Classification
article

Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets

Wannaporn Sarapugdi
article en

Abstract

Comparative machine-learning studies often treat the algorithm with the highest observed score as the preferred model, although that choice may depend on the observations available for training. This study examined model-selection stability under limited training samples across six public classification datasets and five algorithms: Logistic Regression, an RBF-kernel SVM, Random Forest, XGBoost, and LightGBM. For each dataset, 30% of the observations were retained as a fixed evaluation set and the remaining 70% formed a development pool. Training fractions of 10%, 20%, 40%, 60%, 80%, and 100% were evaluated in 30 runs per condition. For fractions below 100%, training composition varied across stratified subsets; the 100% condition reused the complete development pool and served as the reference condition. Macro-F1 was the primary model-selection criterion. Dominant winner rate and normalised selection entropy summarized selection stability, while selection regret compared limited-data choices with the full-data reference. Paired Wilcoxon signed-rank tests with Holm correction were used to assess support for the mean-leading algorithm within the experimental protocol. Within-fraction winner agreement was comparatively high in several datasets, but this did not always mean that the same algorithm remained dominant as more training data became available. Landsat changed from Random Forest at 10–20% to LightGBM from 40% onward, whereas Dry Bean changed from Logistic Regression at 10% to SVM-RBF at intermediate fractions and XGBoost at 80–100%. Mean selection regret declined from 0.0097 to 0.0002 for Landsat and from 0.0058 to 0.0008 for Dry Bean between the 10% and 80% conditions. In five limited-data conditions, the mean-leading algorithm was not statistically supported against all competitors after multiplicity correction. The findings show that both training-sample size and composition can change the selected model and the evidence supporting that choice. They describe model-selection stability under training-sample variation conditional on a single fixed evaluation set.

Journal of Computer and Creative TechnologyVol. 4(3)
Rajabhat Maha Sarakham University (TH)
Machine Learning and Data Classification
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon