MURPHY: improving surgical workflow analysis with large-scale datasets and relation-aware models

Abstract Surgical workflow analysis aims to provide a structured interpretation of operative videos by recognizing procedural steps, tasks, and fine-grained activities (defined as combinations of instruments, actions, and anatomical objects) associated with each video frame. Current supervised methods are limited by the scarcity of large-scale annotated datasets. Moreover, they primarily rely on visual and temporal information, largely overlooking relational cues that describe dependencies within and across workflow levels. To address these gaps, we introduce RLLS12M, a large-scale dataset of robotic left lateral sectionectomies comprising 50 procedures (over 2.1 million frames) performed by five surgeons. The dataset features hierarchical annotations designed to reflect procedural logic and clinically meaningful dependencies. Furthermore, we propose MURPHY, a multi-relation network that leverages these dependencies within and across workflow levels to improve hierarchical recognition. Compared to evaluated baselines, MURPHY demonstrates quantitative improvements, yielding at least +2.99 mAP for activities, +2.26 mAP for tasks, and +1.82 mAP for steps. Ablation and consistency analyses demonstrate that relation-aware components account for the majority of these gains, leading to more coherent predictions that better align with clinical expectations. Although these results underscore the value of relational modeling, a performance gap remains when compared to expert human surgeons, particularly for activity recognition, highlighting a key area for ongoing refinement toward clinical application.