A Survey of Machine Learning Based Systems for Evaluating Expertise Classification in Medical Simulators

Medical simulators provide a safe environment for practising crucial procedures, particularly in virtual simulators where objective and quantitative data can be collected for developing machine learning algorithms for automatic expertise classification. This survey analyses 13 automatic evaluation systems used in medical simulators and identifies best practices for integrating ML algorithms. Among these systems, nine employed commercial simulators, particularly NeuroVR and the Da Vinci robotic systems, while four utilised custom simulators. The survey outlines the main steps in the integration of machine learning algorithms: data collection, metric generation and selection, training, and testing. Metric selection was identified as a crucial factor affecting both the accuracy of the algorithm and the comprehension of the evaluation. Typically, multiple machine learning algorithms were applied to the same dataset to compare results and identify the most effective model. Overall, this survey suggests that transparent algorithms are preferable, as they enhance physicians’ understanding.