Abstention and Threshold Identification for Uncertainty Management in Clinical Decision Tools: A Case Study using Human-In-The-Loop Pediatric Autism Classifiers

Uncertainty quantification remains an underdeveloped aspect of AI-based clinical decision tools. As AI systems become increasingly prevalent in healthcare, it is essential not only to measure uncertainty but also to manage it in ways that support clinical decision-making. In this study, we investigate abstention as a practical mechanism for managing uncertainty in diagnostic classifiers. To stress-test this approach, we deliberately evaluate abstention performance on a purposefully noisy dataset of pediatric autism video assessments comprising heterogeneous video sources and a diverse range of human raters. We apply abstention strategies to existing autism classifiers trained on diagnostic assessment data, comparing baseline performance to a range of thresholding configurations that trade off retained sample coverage against key clinical metrics. We compare performance gains from prioritizing sensitivity or specificity to targeting a balanced increase in Youden's J to demonstrate a wide variety of use cases that abstention can enable. This work demonstrates a concrete use case of introducing abstention into the output range of clinical decision models, enabling both uncertainty quantification and management in diagnostic classifiers.