MMPNeRF: Multi-Modal Neural Rendering of Transparent Objects Using Color and Polarimetric Images

In this work, we propose a methodology that realise novel view synthesis and 3D reconstruction of scene containing transparent object. The transparent objects are hard to perceive, that why we leverage the complementary strengths of two modalities: color and polarimetric imaging. We aim to enhance the reconstruction process by integrating these modalities into a multi-modal Neural Radiance Fields (NeRF) architecture, While RGB data provides crucial information about the visual appearance of transparent objects, polarimetric imaging offers valuable insights into their material properties and surface characteristics. Our method is evaluated and compared to two methods from literature on the creation of a novel view and on the precision of the TSDF estimation on our synthetic dataset. Our results show that the addition of another modality, in this case polarimetry, increases the accuracy of TSDF estimation, and also enables the creation of new, more accurate views of the scene containing transparent objects. Overall, our methodology aims to advance 3D reconstruction by addressing the unique challenges posed by transparent objects, incorporating diverse modalities to achieve more accurate and detailed reconstruction results, thereby enhancing the fidelity and realism of reconstructed scenes.