Model Ranking Stability on Tabular Data: Dataset Size, Carbon Cost, and Fairness

Model comparisons in machine learning are often reported from a single train/test split, one random seed, and one accuracy metric, so rankings may change when any of these choices changes. We asked how much the ranking of a fixed set of models moves under resampling, and whether this depends on dataset size. We ran 500 model fits on ten OpenML tabular classification datasets (270 to 48,842 rows) using five off-the-shelf classifiers (logistic regression, random forest, support vector machine, XGBoost, and multilayer perceptron), two split ratios, and five random seeds. Each run recorded Macro-F1, training time, CodeCarbon emissions and, for four datasets with a usable demographic attribute, the Macro-F1 gap between groups. Ranking stability was measured with Kendall’s τ, the top two models were compared with paired t-tests, and a Pareto front over Macro-F1, emissions, and the fairness gap was computed for each dataset. Mean τ ranged from about zero on some datasets of around 1,000 rows or fewer to above 0.8 on the two largest, and it correlated with dataset size (Spearman’s ρ = 0.79, p = 0.006, 10 datasets, 95% confidence interval 0.33 to 0.95). The top two models differed significantly on 4 of 10 datasets under a pooled paired t-test, on 3 when each split ratio was tested separately, and on 1 after a correction for dependent resamples and a Holm adjustment across datasets. When emissions and the fairness gap were added to Macro-F1, the multilayer perceptron remained Pareto-optimal on 3 datasets, compared with 7 for XGBoost and for the support vector machine.Single-split rankings on small tabular datasets therefore deserve limited confidence, and accuracy-only leaderboards can hide trade-offs. The size association is observational, rests on ten datasets, and covers classical models only. A simple ceiling-effect check detected label leakage in one dataset version.