Towards a Comprehensive Taxonomy of Pointwise Reliability Assessment Techniques for Machine Learning Models

Machine learning techniques have achieved significant success across various domains, including decision-making, speech recognition, natural language processing, and computer vision. However, despite their demonstrated performance, concerns regarding the level of trustworthiness of their predictions are hampering their deployment in critical scenarios. While standard metrics like accuracy, sensitivity, mean error, and coefficient of determination are valuable tools to assess the performance of a machine learning model, they all are inadequate for evaluating individual instances. Moreover, these metrics rely on labelled data and may not generalize well to unseen instances encountered during model deployment. Given these limitations, there is a growing interest in developing measures of reliability and trustworthiness for individual predictions during real-time deployment. Such measures would not only assist in model improvement and development but also enhance trust in human-machine interactions. Ultimately, addressing this challenge is crucial for ensuring that machine learning models can recognize their own limitations and make informed decisions in real-world situations. This study provides a unifying review of the techniques for pointwise reliability assessment, with its primary contribution being the proposal of a taxonomy grounded in the underlying principles and information utilized by these techniques. Additionally, it provides a thorough discussion of related topics and concepts.