Evaluation Metrics for Image Classification, Segmentation and Localization Mathematical Foundations, Comparative Analysis and Research Recommendations

Abstract Evaluation metrics dictate the incentives for an image analysis system, detect remaining failures, and verify any proposed improvement in scientific or clinical claims. Despite the mathematical similarity of classification, segmentation, and localization, the units, matching criteria, and practice implications vary across these tasks. The current review analyzes the distinctions through the lens of a selected sample of 150 bibliographically validated references, including 10 papers from the provided author profile. It incorporates foundational statistics, state-of-the-art benchmarking practices, medical imaging use cases, and recent developments in uncertainties, selective prediction, and deployment. The review organizes 72 metrics and corresponding evaluation quantities by the question they answer: correctness of decision, score discrimination, reliability of probabilities, spatial correspondence, recovery of objects, localization of geometry, or practical utility. Comparisons describe equations, assumptions, aggregation approaches, advantages, and failures. Ten deterministic worked examples demonstrate prevalence effects, constant discrimination and worsening probability scores, reverse rankings because of aggregation, hidden border failures by overlap, detection matching, physical landmark errors, and the cost of abstaining. Twenty-four numeric instances from 4 separate publications supply additional citations and examples without pooling the inconsistent results. The primary takeaway is that a sound evaluation requires a small and preselected set of metrics, complementary to each other, applicable to the defined population, unit, reference standard, and decision. For classification, operating point performance is always required, along with discrimination and probability metrics. Segmentation demands appropriate measures of overlap, borders, and object topology. Localization needs explicitly stated criteria of correspondence and tolerance and false positive control. Independent partitioning, uncertainties, subgroups, and workflow outputs determine the interpretability of the metrics. This manuscript is an organized critical review and methodology resource, but not a completed systematic review according to PRISMA criteria, meta-analysis, or benchmark of a new model. The attached list of metrics, reference export, extraction notes, computational code, and reporting checklist help design and validate reproducible studies and author reviews.