Evaluating the quality of the quantified uncertainty for (Re)calibration of data-driven regression models

• First systematic benchmark of calibration metrics in uncertainty quantification • Calibration metrics frequently produce inconsistent and conflicting evaluations • Using different calibration metrics in regression allows cherry-picking of results • The ENCE and CWC emerge as most reliable metrics for miscalibration detection • Metrics need ≥ 500-1000 samples for robust evaluation in recalibration studies In safety-critical applications data-driven models must not only be accurate but also provide reliable uncertainty estimates. This property, commonly referred to as calibration, is essential for risk-aware decision-making. In regression a wide variety of calibration metrics and recalibration methods have emerged. However, these metrics differ significantly in their definitions, assumptions and scales. As a result, interpreting and comparing results across studies becomes challenging. Moreover, most recalibration methods have been evaluated using only a small subset of metrics, leaving it unclear whether improvements generalize across different notions of calibration. In this work, we systematically extract and categorize regression calibration metrics from the literature. We benchmark these metrics independently of specific modelling methods or recalibration approaches. Through controlled experiments with real-world, synthetic and artificially miscalibrated data, we demonstrate that calibration metrics frequently produce conflicting results. Our analysis reveals substantial inconsistencies: many metrics disagree in their evaluation of the same recalibration result, and some even indicate contradictory conclusions. This inconsistency is particularly concerning as it allows cherry-picking of metrics to create misleading impressions of success. We identify that the Expected Normalized Calibration Error (ENCE) and the Coverage Width-based Criterion (CWC) are the most dependable metrics within our Gaussian uncertainty-based test framework. The results highlight the critical role of metric selection in calibration research.