Real Beats, Below the Margins: A Pre-registered Benchmark in Which Tuned XGBoost and LightGBM Edge the sklearn Credit Champion by Less Than the Frozen Winning Lines, and the Calibration Ranking Is Not the Discrimination Ranking

When a credit team considers replacing a working scorecard library, the question is rarely asked under controlled conditions: the challenger is usually tuned, the incumbent usually is not, and the comparison is scored on one ranking metric. This study froze the question before measuring it. Under a pre-registration locked before any measured run, XGBoost 3.4.1 and LightGBM 4.7.0 were benchmarked against the published sklearn gradient-boosting champion of a companion study (credit-risk-lab) on that study's own leakage-safe protocol: the UCI default of credit card clients data (30,000 accounts), a stratified 70/30 split at seed 42 (21,000 training rows, 9,000 test rows), preprocessing fit on training rows only, and every arm scored on the identical preprocessed matrix. Each tuned arm received exactly 24 random-search trials under the same stratified 3-fold cross-validation on the training rows, with selection by mean cross-validation AUC and the test split scored exactly once per final model. A sixth arm, the champion's own library given the same 24-trial budget, separated the library claim from the tuning claim. The reproduction gate passed exactly: the control arm reproduced the published champion figures to the last printed digit (AUC 0.7810, KS 0.4196, Brier 0.1355, delta 0.0000 on all three). H1 is INCONCLUSIVE: tuned XGBoost beat the champion on AUC, 0.7822 against 0.7810, a delta of +0.0012, but the frozen margin for a confirmed beat was +0.005. H2 is INCONCLUSIVE: tuned LightGBM beat the champion on KS, 0.4239 against 0.4196, a delta of +0.0043, against a frozen margin of +0.010. H3 is CONFIRMED: the calibration ranking is not the discrimination ranking; by AUC the order is XGBoost, LightGBM, champion, and by Brier it is LightGBM, XGBoost, champion, so the AUC winner is not the calibration winner. With equal tuning budgets the three tuned libraries land within 0.0010 AUC of each other (0.7812 to 0.7822), which is the study's central measurement: on this protocol the champion's deficit to its tuned rivals is a tuning deficit, not a library deficit. Library defaults tell the opposite story to tuning: XGBoost at defaults trails the champion badly (AUC 0.7598), while LightGBM at defaults already beats the champion on KS (0.4237 against 0.4196). Which library wins has no answer until the tuning budget is named, and a swap chosen on the ranking metric alone would have picked the worse-calibrated of the two challengers.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328939
Primary Topic
Financial Distress and Bankruptcy Prediction
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Real Beats, Below the Margins: A Pre-registered Benchmark in Which Tuned XGBoost and LightGBM Edge the sklearn Credit Champion by Less Than the Frozen Winning Lines, and the Calibration Ranking Is Not the Discrimination Ranking

Jesun Ahmad Ushno
2 links
Zenodo (CERN European Organization for Nuclear Research)
Financial Distress and Bankruptcy Prediction
preprint

Real Beats, Below the Margins: A Pre-registered Benchmark in Which Tuned XGBoost and LightGBM Edge the sklearn Credit Champion by Less Than the Frozen Winning Lines, and the Calibration Ranking Is Not the Discrimination Ranking

Jesun Ahmad Ushno
preprint en

Abstract

When a credit team considers replacing a working scorecard library, the question is rarely asked under controlled conditions: the challenger is usually tuned, the incumbent usually is not, and the comparison is scored on one ranking metric. This study froze the question before measuring it. Under a pre-registration locked before any measured run, XGBoost 3.4.1 and LightGBM 4.7.0 were benchmarked against the published sklearn gradient-boosting champion of a companion study (credit-risk-lab) on that study's own leakage-safe protocol: the UCI default of credit card clients data (30,000 accounts), a stratified 70/30 split at seed 42 (21,000 training rows, 9,000 test rows), preprocessing fit on training rows only, and every arm scored on the identical preprocessed matrix. Each tuned arm received exactly 24 random-search trials under the same stratified 3-fold cross-validation on the training rows, with selection by mean cross-validation AUC and the test split scored exactly once per final model. A sixth arm, the champion's own library given the same 24-trial budget, separated the library claim from the tuning claim. The reproduction gate passed exactly: the control arm reproduced the published champion figures to the last printed digit (AUC 0.7810, KS 0.4196, Brier 0.1355, delta 0.0000 on all three). H1 is INCONCLUSIVE: tuned XGBoost beat the champion on AUC, 0.7822 against 0.7810, a delta of +0.0012, but the frozen margin for a confirmed beat was +0.005. H2 is INCONCLUSIVE: tuned LightGBM beat the champion on KS, 0.4239 against 0.4196, a delta of +0.0043, against a frozen margin of +0.010. H3 is CONFIRMED: the calibration ranking is not the discrimination ranking; by AUC the order is XGBoost, LightGBM, champion, and by Brier it is LightGBM, XGBoost, champion, so the AUC winner is not the calibration winner. With equal tuning budgets the three tuned libraries land within 0.0010 AUC of each other (0.7812 to 0.7822), which is the study's central measurement: on this protocol the champion's deficit to its tuned rivals is a tuning deficit, not a library deficit. Library defaults tell the opposite story to tuning: XGBoost at defaults trails the champion badly (AUC 0.7598), while LightGBM at defaults already beats the champion on KS (0.4237 against 0.4196). Which library wins has no answer until the tuning budget is named, and a swap chosen on the ranking metric alone would have picked the worse-calibrated of the two challengers.

Zenodo (CERN European Organization for Nuclear Research)
Financial Distress and Bankruptcy Prediction
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon