Automated machine learning assessment and feedback for the Fundamentals of Laparoscopic Surgery peg transfer task

Aim: Whereas computer vision has advanced the capabilities of Fundamentals of Laparoscopic Surgery (FLS) trainers, real-time coaching using machine learning (ML) remains limited. This retrospective proof-of-concept study evaluated whether an ML model could provide automated FLS pass/fail assessment comparable to faculty, using pre-recorded videos. Methods: Residents performed video-recorded FLS peg transfers scored pass/fail. A training set of 60 videos (51 standard cases, 9 edge-case videos) was used to develop a multi-stage pipeline combining a “you only look once” (YOLOv8n) object detector, and a deterministic rule engine encoding FLS pass/fail criteria. The model was evaluated on 24 resident-disjoint, held-out videos. A faculty majority-vote reference standard was set by five evaluators per video. Model predictions were compared against this reference standard. Results: The ML model classified outcomes accurately in 92% of cases [95%CI: 74%-98%; F1 score = 0.93 (pass positive); Cohen’s κ = 0.83]. Total analysis time was 6 min vs. approximately 56 min of video playback. The model concurrently identified errors in 70% (31/44) of error events, with 97% (30/31) concordance in error classification. Among the 13 remaining events, the model preceded the faculty-annotated timestamp in 8/13 (62%). Real-time instructional prompts were implemented in a heads-up display but not evaluated here. Conclusion: The system provided accurate FLS peg-transfer assessment comparable to faculty while reducing evaluation time. As a retrospective proof-of-concept, it supports automated assessment and, pending prospective validation, could deliver scalable real-time feedback in skills training. External validation in larger, multi-institutional cohorts should precede adoption.

Authors

Institutions

Publication Details

Journal
Artificial Intelligence Surgery
Published
2026-11-19
DOI
https://doi.org/10.20517/ais.2026.72
Primary Topic
Surgical Simulation and Training
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Automated machine learning assessment and feedback for the Fundamentals of Laparoscopic Surgery peg transfer task

Issam Koleilat, Philip S. Barie, Joelle Getrajdman, Mayur Narayan et al.
Artificial Intelligence Surgery
Surgical Simulation and Training
article

Automated machine learning assessment and feedback for the Fundamentals of Laparoscopic Surgery peg transfer task

Issam Koleilat, Philip S. Barie, Joelle Getrajdman, Mayur Narayan, Advaith Bongu, Amin Madani, Yunzhe Xue, Usman Roshan, Divya Kewalramani, Nell Maloney Patel, Andrew Hu, Finn Kliewer, Jenny Cai, Ricardo Nunez
article en

Abstract

Aim: Whereas computer vision has advanced the capabilities of Fundamentals of Laparoscopic Surgery (FLS) trainers, real-time coaching using machine learning (ML) remains limited. This retrospective proof-of-concept study evaluated whether an ML model could provide automated FLS pass/fail assessment comparable to faculty, using pre-recorded videos. Methods: Residents performed video-recorded FLS peg transfers scored pass/fail. A training set of 60 videos (51 standard cases, 9 edge-case videos) was used to develop a multi-stage pipeline combining a “you only look once” (YOLOv8n) object detector, and a deterministic rule engine encoding FLS pass/fail criteria. The model was evaluated on 24 resident-disjoint, held-out videos. A faculty majority-vote reference standard was set by five evaluators per video. Model predictions were compared against this reference standard. Results: The ML model classified outcomes accurately in 92% of cases [95%CI: 74%-98%; F1 score = 0.93 (pass positive); Cohen’s κ = 0.83]. Total analysis time was 6 min vs. approximately 56 min of video playback. The model concurrently identified errors in 70% (31/44) of error events, with 97% (30/31) concordance in error classification. Among the 13 remaining events, the model preceded the faculty-annotated timestamp in 8/13 (62%). Real-time instructional prompts were implemented in a heads-up display but not evaluated here. Conclusion: The system provided accurate FLS peg-transfer assessment comparable to faculty while reducing evaluation time. As a retrospective proof-of-concept, it supports automated assessment and, pending prospective validation, could deliver scalable real-time feedback in skills training. External validation in larger, multi-institutional cohorts should precede adoption.

Artificial Intelligence SurgeryVol. 6(4)
Johnson University (US)
Surgical Simulation and Training
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon