Software Defect Prediction evaluation: New metrics based on the ROC curve

Context: ROC (Receiver Operating Characteristic) curves are widely used to represent how well fault-proneness models (e.g., probability models) classify software modules as faulty or non-faulty. AUC, the Area Under the ROC Curve, is usually used to quantify the overall discriminating power of a fault-proneness model. Alternative indicators proposed, e.g., RRA (Ratio of Relevant Areas), consider the area under a portion of a ROC curve. Each point of a ROC curve represents a binary classifier, obtained by setting a specified threshold on the fault-proneness model. Several performance metrics (Precision, Recall, the F-score, etc.) are used to assess a binary classifier. Objectives: We investigate the relationships linking ‘‘under the ROC curve area’’ indicators such as AUC and RRA to performance metrics. Methods: We study these relationships analytically. We introduce iso-PM ROC curves, whose points have the same value isoPM for a given performance metric PM. When evaluating a ROC curve, we identify the iso-PM curve with the same value of AUC or RRA. Its isoPM can be seen as a property of the ROC curve and fault-proneness model under evaluation. Results: There is an S-shaped relationship between isoPM and AUC for performance metrics that do not depend on the proportion ρ of faulty modules, i.e., dataset balancedness. φ (Matthews Correlation Coefficient) depends on ρ: with very imbalanced datasets, AUC appears over-optimistic and φ over-pessimistic. RRA defines the region of interest in terms of ρ, so all performance metrics depend on ρ. RRA is related to performance metrics via S-shaped curves. Conclusion: Our proposal helps gain a better quantitative understanding of the goodness of a ROC curve, especially in practically relevant regions of interest. Also, showing a ROC curve and iso-PM curves provides an intuitive perception of the goodness of a fault-proneness model.