ROC analysis for clinical decision support systems (CDSS) results in digital mammography images

Artificial intelligence and clinical decision support systems (CDSS) are being actively implemented in healthcare. Radiology is at the forefront of the use of such technologies. In this article, we describe a method for evaluating the performance of CDSS, including software based on artificial intelligence technologies (AI-based software), which is suitable for any medical organization that needs to assess the applicability of such software. Purpose: The purpose of this study is to demonstrate the use of a web-based ROC analysis tool for evaluating the performance of clinical decision support systems (CDSS) using digital mammography images as an example. Materials and methods: A retrospective dataset of mammography studies was used, based on the results of the calibration test report during the version change of one of the AI service participating in the Experiment on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow, with versions of the AI service dated 15.02.2023 and 30.05.2023. The sample size consisted of 100 trials. In this publication, ROC analysis implemented using a web-based tool will be used to evaluate the results of the AI service. Results: The functionality of a web-based tool for ROC analysis was demonstrated using the example of evaluating the performance of AI-based software for processing digital mammography images. Conclusion: By using the presented web-based ROC analysis tool, the verification of СDSS, including AI-based software, as well as the assessment of its performance, can be performed without the need for additional tools if necessary.