Evaluation of the effectiveness of a medical decision support system for preliminary diagnosis in Moscow outpatient clinics

The implementation of the medical decision support system (MDSS) in clinical practice requires careful monitoring to ensure patient safety and track the performance of artificial intelligence technologies.Goal: To assess the effectiveness of the TOP-3 MDSS in outpatient clinics of the Moscow Heath Care Department.Materials and methods: The Moscow Heath Care Department monitored the TOP-3 operation between October 1, 2020 and March 03, 2024 (n = 63,809,360 people). The Hit-3 metric was used to determine if the MDSS needs retraining. An additional study involving medical experts included a retrospective analysis of data from 3,000 patients and calculation of the diagnostic agreement rate among the MDSS, doctors and the experts.Results: The monitoring demonstrated mean Hit-3 of 63.5, 64.5 and 67.7 for MDSS ver. 1, 2, and 3, respectively. In cases where the clinicians disagreed with the MDSS (n = 2000), based on the patient complaints, experts agreed with the MDSS in 80.2% of cases, with clinicians in 11.5% of cases, and established a different diagnosis in 8.3% of cases. In cases where the clinicians’ and the MDSS’s conclusions matched (n = 1000), experts approved the diagnosis in 50.4% of cases, selected an alternative diagnosis suggested by the MDSS in 37.9% of cases, and established a different diagnosis for 11.7% of patients. Conclusion: The described monitoring methodology, supplemented by expert review, allowed for a comprehensive assessment of the MDSS to be implemented in the health care system. Based on the results of the TOP-3 effectiveness assessment, it was decided that the analyzed list of data from electronic health records should be expanded, which will be implemented in the upcoming version TOP-3+.