Toward comprehensive real-time scene understanding in ophthalmic surgery through multimodal image fusion

Abstract Purpose The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery. Methods We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence. Results Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from $$284\,\upmu \hbox {m}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>284</mml:mn> <mml:mspace/> <mml:mi>μ</mml:mi> <mml:mtext>m</mml:mtext> </mml:mrow> </mml:math> (OPMI only) to $$33\,\upmu \hbox {m}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>33</mml:mn> <mml:mspace/> <mml:mi>μ</mml:mi> <mml:mtext>m</mml:mtext> </mml:mrow> </mml:math> (multimodal). Conclusion Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.