An Evaluation Method for Machine Learning Based on Verification and Correction Costs

We propose a new method to evaluate machine learning models, parameters, and algorithms. To design themethod, we consider verification and correction costs of human labor, and we base it on machine learning systemswith reject models. The models are proposed to handle problems where the machine learning systems have errors.In the processes of the reject models, a part of outputs of machine learning are rejected and we can get our desiredaccuracy, that means we can reduce errors, for the other results that are not rejected. On the other hand, we shouldverify and correct the rejected results. Therefore, the verification and correction costs should be considered to developthe evaluation method. In addition, the reject models are sensitive to thresholds that define decisions of rejection.Thus, the evaluation method should handle the varying thresholds. Hence, the method to evaluate the machinelearning should have the following two features: (1) handling varying thresholds, (2) managing the verification andcorrection costs. Conventional methods such as ROC curve or PR curve can handle the varying thresholds. Theseconventional methods, however, cannot manage the verification and correction costs. In this paper, first, we define aperformance measure to evaluate machine learning based on the verification and correction costs. Second, we proposeARAC curve(Acceptance Rate-Accuracy after Correction curve) and ARAC-AUC(Area Under Curve) in which thehorizontal axis shows acceptance rate, and the vertical axis shows accuracy after correction, respectively. This ARACcurve can handle varying thresholds as well as the conventional methods. Third, we explain the relationship theperformance measure and the ARAC curve. The horizontal axis is closely related to the verification costs, andthe vertical axis is closely related to the correction costs. Accordingly, the ARAC curve can express verificationand correction costs. Finally, we show experimental results where the proposed ARAC curve and ARAC-AUC canexpress the performance measure better than the conventional methods.