EndoMODE: A Multimodal Visual Feature-Based Ego-Motion Estimation Framework for Monocular Odometry and Depth Estimation in Various Endoscopic Scenes

Ego-motion estimation is a critical task for both monocular odometry and depth estimation, which are significant for navigation and scene perception in endoscopy. Most of existing ego-motion estimation frameworks only utilize a single modality of visual feature. In this work, a novel multimodal visual feature-based framework is proposed to perform real-time vision-based ego-motion estimation for visual odometry and depth estimation in endoscopic scenes. In the framework, to extract correlation information of two adjacent endoscopic frames, a channel attention-based module is proposed to integrate multidimensional features from the concatenated image and a feature interaction module is proposed to fuse scene features from two observations. Moreover, a novel pose decoder based on depthwise separable convolution is proposed to extract multiscale feature representation. Furthermore, a fully supervised pipeline and a self-supervised pipeline with the proposed framework are designed for monocular odometry and depth estimation, respectively. In the experiments, the proposed framework is compared with several advanced methods on five different datasets of various endoscopic scenes. Both quantitative and qualitative results demonstrate the pipeline with the proposed framework provides the most accurate monocular odometry and depth estimation in different kinds of endoscopic scenes.