Color-polarimetric stereo endoscopy for high-quality 3D reconstruction

3D endoscopy can improve minimally invasive diagnosis and surgery by enhancing the precision, safety, and efficiency of surgery and enabling computer-assisted interventions. However, traditional approaches relying on feature detection and stereo matching often struggle with the weakly textured features of biological tissues and confusing artifacts such as specular reflections, leading to compromised reconstruction quality. Here, we demonstrate color-polarimetric stereo endoscopy, which leverages the dense surface normal information carried by polarization to enhance feature-based reconstruction from binocular color imaging. This approach was enabled by developing a non-depolarizing binocular endoscope, which is suitable for snapshot color-polarimetric image acquisition, and a deep neural network for seamless fusion of polarimetric and binocular color information for 3D reconstruction of biological tissues. The proposed approach was validated with tissue phantoms and ex vivo porcine tissues. The results indicate that it outperforms reconstruction methods based solely on binocular color cues, including techniques based on learning and block matching. In regions showing weak textures and specular reflections, reconstruction errors were reduced by 29.67% and 13.24%, respectively. The work provides a solution to overcoming the limitations of current state-of-the-art 3D endoscopy techniques, paving the way for intelligent and precise minimally invasive procedures.