Evaluating the Performance of AI Models Under Multiple Criteria

AbstractThe evaluation of a model performance has traditionally been performed by staff with quantitative skills. As organizations increasingly rely on AI models to support business applications or to inform their decisions, more stakeholders will undeniably be interested in whether the performance of these models is meeting their individual criteria. In Salami et al (2023) [1], we considered two stakeholders including the model owner1and the model developer2 having two different success criteria; and we showed that linking the performance metrics they care about can enable a more effective evaluation of the model during the monitoring phase. In this paper, we propose to expand this linkage to the development phase to identify which model meetsthese two success criteria hence should be selected for deployment. Moreover, we propose a methodology to optimize the relationship between the performance metrics of these two stakeholders. Finally, we show how the lack of understanding of the business context can lead to the selection of an inadequate statistical performance metric, which can lead to erroneous decisions with severely adverse business impact depending on the model applications