The Model That Wins on AUC Is Not the Model That Saves the Bank Money: A Cost-Sensitive Re-Evaluation of Credit Default Prediction Classifiers

Published benchmarks for credit-card default prediction almost universally rank classifiers by rank-based or threshold-free metrics such as AUC, F1, or accuracy, implicitly treating a missed default and a wrongly rejected good customer as equally costly errors. In practice, a bank's loss from an undetected default (loss given default on the full credit exposure) is an order of magnitude larger than its opportunity loss from rejecting a creditworthy applicant (one month of foregone interest margin), so a model ranking built on symmetric-cost metrics need not reflect which model actually minimizes the bank's expected loss. We revisit the UCI "Default of Credit Card Clients" benchmark (30,000 accounts, Taiwan, 2005) - a dataset underlying dozens of published single-split classifier comparisons - using a paired, Optuna-tuned, repeated stratified cross-validation protocol (5-fold x 6 repeats, 30 paired folds) across Logistic Regression, Random Forest, XGBoost, and LightGBM. Each model is evaluated twice: once under the conventional 0.5 decision threshold with AUC/F1/MCC, and once under a nested, leakage-free, cost-optimized decision threshold derived from an exposure-weighted cost matrix built directly from each account's own credit limit. Friedman and post-hoc Nemenyi tests show LightGBM is the significantly best-ranked model by AUC (p<0.001 vs. Random Forest), yet Random Forest attains the lowest mean expected cost (1.24% of portfolio exposure), with no statistically significant cost difference among the three tree ensembles after correction for multiple comparisons - the model ranking flips depending on which metric is used to read the same cross-validation folds. A far larger and more consistent effect, however, is threshold calibration itself: replacing the default 0.5 cutoff with a cost-optimized threshold cuts expected cost by 86-88% for every model tested (Wilcoxon p<10^-8), an effect that dwarfs the 1-6 percentage-point AUC spread the literature treats as the interesting comparison. These results suggest that for applied credit-scoring work, calibrating the decision threshold to the institution's own cost structure delivers more value than choosing among competitive ensemble algorithms, and that AUC-based leaderboards can be actively misleading about which model a bank should deploy.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24304545
Primary Topic
Financial Distress and Bankruptcy Prediction
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Model That Wins on AUC Is Not the Model That Saves the Bank Money: A Cost-Sensitive Re-Evaluation of Credit Default Prediction Classifiers

Nezar Abdilah Prakasa
2 links
Zenodo (CERN European Organization for Nuclear Research)
Financial Distress and Bankruptcy Prediction
preprint

The Model That Wins on AUC Is Not the Model That Saves the Bank Money: A Cost-Sensitive Re-Evaluation of Credit Default Prediction Classifiers

Nezar Abdilah Prakasa
preprint en

Abstract

Published benchmarks for credit-card default prediction almost universally rank classifiers by rank-based or threshold-free metrics such as AUC, F1, or accuracy, implicitly treating a missed default and a wrongly rejected good customer as equally costly errors. In practice, a bank's loss from an undetected default (loss given default on the full credit exposure) is an order of magnitude larger than its opportunity loss from rejecting a creditworthy applicant (one month of foregone interest margin), so a model ranking built on symmetric-cost metrics need not reflect which model actually minimizes the bank's expected loss. We revisit the UCI "Default of Credit Card Clients" benchmark (30,000 accounts, Taiwan, 2005) - a dataset underlying dozens of published single-split classifier comparisons - using a paired, Optuna-tuned, repeated stratified cross-validation protocol (5-fold x 6 repeats, 30 paired folds) across Logistic Regression, Random Forest, XGBoost, and LightGBM. Each model is evaluated twice: once under the conventional 0.5 decision threshold with AUC/F1/MCC, and once under a nested, leakage-free, cost-optimized decision threshold derived from an exposure-weighted cost matrix built directly from each account's own credit limit. Friedman and post-hoc Nemenyi tests show LightGBM is the significantly best-ranked model by AUC (p<0.001 vs. Random Forest), yet Random Forest attains the lowest mean expected cost (1.24% of portfolio exposure), with no statistically significant cost difference among the three tree ensembles after correction for multiple comparisons - the model ranking flips depending on which metric is used to read the same cross-validation folds. A far larger and more consistent effect, however, is threshold calibration itself: replacing the default 0.5 cutoff with a cost-optimized threshold cuts expected cost by 86-88% for every model tested (Wilcoxon p<10^-8), an effect that dwarfs the 1-6 percentage-point AUC spread the literature treats as the interesting comparison. These results suggest that for applied credit-scoring work, calibrating the decision threshold to the institution's own cost structure delivers more value than choosing among competitive ensemble algorithms, and that AUC-based leaderboards can be actively misleading about which model a bank should deploy.

Zenodo (CERN European Organization for Nuclear Research)
Financial Distress and Bankruptcy Prediction
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon