Surgeon-in-the-loop: expert-guided machine learning framework for continuous surgical performance assessment using operative video

A surgeon’s level of technical proficiency plays a significant role in surgical outcomes. Rating scales, such as the objective structured assessment of technical skills (OSATS), have been validated for surgical quality assurance, skills assessment, and training. However, they are subjective, time-consuming, and dependent on experts with busy schedules, limiting scalability. Moreover, experts may face a long delay between procedure completion and their single, unitary rating, preventing continuous feedback on actions performed. Artificial intelligence systems trained on expert ratings may help overcome these limitations. This work demonstrates a comprehensive human-in-the-loop framework for developing a machine-learning system for automated, continuous OSATS ratings, from data collection to algorithm training and real-world applications. We provide demonstrations of AI assessment on two video recordings: (1) removal of an epileptogenic region during intracranial surgery, and (2) resection of a simulated subpial brain tumor. With sufficient training and validation, machine-learning systems may approximate expert assessments and provide real-time, expert-level feedback during procedures to inform surgeons of increasing patient risk and improve operative outcomes.