Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Abstract Reliable evaluation of supervised machine learning models requires more than reporting a standard set of performance metrics. Although extensive guidance exists on individual metrics and validation techniques, this knowledge is often considered in isolation, making it difficult to determine which evaluation choices are appropriate under different data characteristics and operational objectives. This review and tutorial addresses this gap by presenting a unified, decision-oriented perspective on model evaluation that connects prediction task, dataset structure, error consequences, validation design, and metric selection. Rather than providing only a catalogue of evaluation measures, the paper combines methodological discussion with controlled empirical scenarios across diverse classification and regression datasets to demonstrate when and why commonly used metrics can lead to different interpretations of model performance. The scenarios examine class imbalance, asymmetric misclassification costs, multiclass averaging, probability calibration, outlier sensitivity, residual behaviour, and the instability of percentage-based errors. Particular attention is given to cases in which apparently strong performance under one criterion conceals important weaknesses under another. Through these examples, the paper provides practical guidance for selecting and interpreting complementary metrics and validation strategies according to the statistical properties of the data and the consequences of prediction errors. The resulting framework positions model evaluation as a context-dependent decision process rather than a post hoc reporting exercise, providing practitioners and researchers with a structured basis for designing more reliable, transparent, and operationally meaningful evaluations of supervised machine learning models.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11283074/v1
Citations
1
Primary Topic
Imbalanced Data Classification Techniques
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Marcello Trovati, Nikolaos Polatidis, I Martin, Xuanyan Liu et al.
1 citations
Research Square
Imbalanced Data Classification Techniques
preprint

Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Marcello Trovati, Nikolaos Polatidis, I Martin, Xuanyan Liu, Xiaolong Xu
preprint en
1 citations

Abstract

Abstract Reliable evaluation of supervised machine learning models requires more than reporting a standard set of performance metrics. Although extensive guidance exists on individual metrics and validation techniques, this knowledge is often considered in isolation, making it difficult to determine which evaluation choices are appropriate under different data characteristics and operational objectives. This review and tutorial addresses this gap by presenting a unified, decision-oriented perspective on model evaluation that connects prediction task, dataset structure, error consequences, validation design, and metric selection. Rather than providing only a catalogue of evaluation measures, the paper combines methodological discussion with controlled empirical scenarios across diverse classification and regression datasets to demonstrate when and why commonly used metrics can lead to different interpretations of model performance. The scenarios examine class imbalance, asymmetric misclassification costs, multiclass averaging, probability calibration, outlier sensitivity, residual behaviour, and the instability of percentage-based errors. Particular attention is given to cases in which apparently strong performance under one criterion conceals important weaknesses under another. Through these examples, the paper provides practical guidance for selecting and interpreting complementary metrics and validation strategies according to the statistical properties of the data and the consequences of prediction errors. The resulting framework positions model evaluation as a context-dependent decision process rather than a post hoc reporting exercise, providing practitioners and researchers with a structured basis for designing more reliable, transparent, and operationally meaningful evaluations of supervised machine learning models.

Research Square
University of Lancashire (GB), Nanjing University of Posts and Telecommunications (CN), University of Brighton (GB)
European Commission, HORIZON EUROPE Framework Programme
Imbalanced Data Classification Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon