SurgFC: Multimodal Surgical Function Calling Framework on the Demand of Surgeons

The surgical intervention is crucial to patient healthcare, and many studies have developed advanced algorithms to provide understanding and decision-making assistance for surgeons. Despite great progress, these algorithms are developed for a single specific task and scenario, and in practice require the manual combination of different functions, thus limiting the applicability. Thus, an intelligent surgical assistant is expected to accurately understand the surgeon’s intentions and accordingly conduct the specific tasks to support the surgical process. In this work, by improving advanced multimodal large language models (MLLMs), we propose a multimodal Surgical Function Calling (SurgFC) framework that can accurately understand the surgeon’s intention and complete a series of surgical understanding tasks, e.g., surgical scene analysis, surgical instrument detection, and segmentation on demand. Specifically, to achieve superior surgical multimodal understanding, we devise a mixture-of-projectors (MOP) module to align the surgical MLLM in SurgFC to balance the natural and surgical knowledge. Moreover, we devise a surgical Function-Calling Tuning strategy to enable the SurgFC to understand surgical intentions, and thus make a series of surgical function calls on demand to meet the needs of the surgeons. Extensive experiments on neurosurgery data confirm that our SurgFC can understand the surgeon’s intention more accurately than the existing MLLM, resulting in overwhelming performance in textual analysis and visual tasks. The source code is available at https://github.com/franciszchen/SurgFC.