Why is Data Misclassified? Quantifying Data Difficulty for Machine Learning Models with Dual Probability Difficulty Measure

Despite the high accuracy of machine learning models, misclassification of data may raise issues regarding the trustworthiness of these models. There are a number of reasons for misclassification, such as model complexity and the difference between the distributions of training data and evaluation data. In this paper, we investigate whether specific data could be hard to classify correctly based on the training dataset or not. The development of measures to quantify the difficulty of data provides insight into assessing the potential performance of such models. However, most of the existing data difficulty measures, which are typically model-based, suffer from various model biases, while the few existing model-agnostic measures do not yield consistent results across datasets and machine learning models. In this paper, we propose a novel model-agnostic measure named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dual Probability Difficulty Measure</i>. This measure considers the probability of the data instance label and the probability of the most frequent label that does not share the same label as the data instance in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i>-nearest neighbors to assess the data difficulty. We conducted experiments to evaluate the accuracy of machine learning models with respect to different data difficulty levels and utilized the area under the curve to assess the effectiveness of our proposed method compared to other methods. Our experimental results on five diverse datasets across various machine learning models show that our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dual Probability Difficulty Measure</i> has a correlation with model performance and identifies data likely to be misclassified.