Endo-PairGS: pair priors for dynamic endoscopic scene reconstruction

PURPOSE: Dynamic scene reconstruction in endoscopic data is crucial for minimally invasive surgery and surgical navigation. Recently, Gaussian splatting (GS)-based methods have performed outstandingly in the reconstruction of dynamic endoscopic scenes. However, more challenging scenes with deformable tissues, surgical tool occlusion, and changeable camera positions make it harder to initialize a GS model well. To address this, we propose Endo-PairGS, which leverages aligned point cloud pairs from two different frames to initialize a 4D GS model. METHODS: Our framework contains two parts: static and dynamic 3D reconstruction. Firstly, we fine-tune a foundation model on endoscopic data to obtain aligned point clouds. To ensure the stability of fine-tuning video sequences with deformable tissues, we use an optical flow-based mask to reduce the influence of regions with large movement. Then, we used the generated point cloud pairs to initialize a 4D GS model for dynamic scene reconstruction. The paired point clouds can complement the region masked by the surgical tool and the unseen region caused by the camera position change, which presents a more complete 3D scene. RESULTS: The experiments on EndoNeRF and StereoMIS datasets demonstrate our Endo-PairGS outperformed existing methods on both quantitative and qualitative results. Our method got 32.47 and 0.871 on PSNR and SSIM in StereoMIS (P3), improving 4% and 4.8% compared to the baseline approach, respectively. CONCLUSION: This study proposed Endo-PairGS with a more complete initialization method, significantly improving the performance of dynamic endoscopic scene reconstruction. The codes and data will be released at https://github.com/MoriLabNU/Endo-PairGS .