Multinomial Classification Certainty: a new uncertainty metric for multinomial outcome prediction

Abstract Certainty of classifications is crucial when it comes to the practical application of machine learning models. Model performance measures such as accuracy are focused on the average performance of a model. However, when a model is used in a practical setting, such as a medical clinic, it is more important to know how certain the model is of a given prediction or classification than its average performance. Unfortunately, often models only provide a final classification label, usually of the class with the highest probability. This output, however, is not sufficiently informative of the certainty of this particular classification, especially in the presence of multiple classes: the highest probability might be only barely higher than the second highest. Even when a probability distribution is provided, there is no established metric to determine if a particular classification is more certain than a different one. In this article we propose a novel metric we have termed Multinomial Classification Certainty, to represent the certainty of model predictions. We discuss why existing methods cannot represent this type of certainty and we show the mathematical meaning behind important thresholds for this new measure.