Calibrated kNN classification via second-layer neighborhood analysis

Abstract The integration of artificial intelligence (AI) into medical practices has emphasized the importance of ensuring the reliability of predictive confidence, as it influences decision-making and the efficacy of AI-driven solutions. This paper focuses on the utilization of a non-parametric machine learning method, with a particular focus on the computation of confidence scores for individual classifications. Unlike parametric approaches, non-parametric techniques like k-nearest neighbors (kNN) offer flexibility without imposing strict data distribution assumptions. Leveraging this flexibility, we propose a novel kNN approach that introduces confidence-awareness through a two-layered neighborhood analysis. The developed approach is intended to support the classical non-parametric kNN classifier by providing more reliable and trustworthy class probabilities. Experimental evaluations conducted on benchmark datasets as well as a de-identified clinical real-world Electronic Health Records (EHR) data table consisting of thousands of unique class labels demonstrate the effectiveness of our approach in enhancing both, prediction accuracy and certainty assessment.