A Rank Graduation Box for SAFE AI

The growth of Artificial Intelligence applications requires to develop risk management models that can balance opportunities with risks. We contribute to the development of Artificial Intelligence risk models proposing a Rank Graduation Box (RGB), a set of integrated statistical metrics that can measure the “Sustainability”, “Accuracy”, “Fairness” and “Explainability” of any Artificial Intelligence application. Our metrics are consistent with each other, as they are all derived from a common underlying statistical methodology: the Lorenz curve. The validity of the metrics is assessed by means of their practical application to both simulated and real data. The results from the comparison of alternative machine learning models to simulated data are aligned with the generating models, in general indicating linear regression models as the most accurate, regression tree models as the most fair, Random Forest models as the most robust; and leading to model explanations similar to the true ones from the generating model. The outcomes from the application of Random Forest models to real data show that the proposed RGB metrics are more interpretable and more consistent with the expectations, with respect to standard metrics such as AUC, RMSE and Shapley values. The evidence also shows that the RGB metrics are very general and can be applied to any machine learning method, regardless of the underlying data and model. • A new toolkit for evaluating the trustworthiness condition of AI systems. • Our method is based on the use of the Lorenz and concordance curves. • The metrics provide a measure of the compliance AI applications as a percentage. • The application shows the suitability of the approach for assessing AI methods.