Uncertainty quantification of Machine Learning models with applications in Healthcare

Accurately quantifying the uncertainty of a machine learning (ML) model’s prediction enables the identification and proper management of cases in which the model is likely to be wrong. This is crucial in high-stakes decision-making pipelines, where prediction errors can have serious consequences. For example, in the prehospital triage of acute coronary syndrome (ACS), a false negative can delay life-saving treatment, whereas a false positive can lead to wasteful use of limited clinical resources. Motivated by the goal of reducing prediction errors, this dissertation develops new methods for estimating uncertainty and for improving existing approaches. Specifically, this thesis introduces GBDT+SC, an ML model that achieves the best reported rule-out performance for prehospital ACS by identifying the subpopulation suitable for automatic classification. It also presents CC, an algorithm that controls the false positive rate and false negative rate of pre-estimated risk scores—such as sparse linear models with integer coefficients—by adapting risk stratification thresholds to the deployment population. In addition, it develops EWMV, a computationally tractable set-level conformal aggregation algorithm that preserves validity and improves efficiency across a variety of conformal methods and tasks by leveraging calibration data and a collection of pre-trained models. Lastly, it presents REWMV, an extension of EWMV to improve the efficiency of prediction intervals for regression. Together, these methods improve the practical utility of both proposed and established uncertainty estimation approaches, while helping bridge the gap between the desired rule-out performance for ACS and that achieved by existing predictors.