Visually Impaired People Recognize Virtual Objects Through 3D Reconstructed Shapes

With the improvement of three-dimensional (3D) reconstruction technology, virtual objects with realistic shapes have opened up the possibility for the blind or visually impaired (BVI) to improve the cognitive ability of objects by perceiving a large number of objects. In order to help BVI learn and recognize objects more conveniently, we constructed a virtual object shape recognition system (VOSRS) based on 3D reconstruction. We first proposed an improved 3D Gaussian Splatting (3DGS) method for precise 3D reconstruction of objects. Then, we introduced a mesh extraction method for 3D Gaussian to obtain a smooth 3D mesh model suitable for touch perception. Finally, we performed haptic rendering on the 3D mesh model and built an interactive system. We introduced the object 3D reconstruction process and mesh extraction steps in detail, and investigated the effect of object shape cognitive learning using the constructed system through user experiments. We conducted several experiments on the NeRF Synthetic dataset and the DTU dataset. The experimental results show that the improved 3DGS method can effectively reduce spikes and artifacts, speed up the iteration process, and improve the reconstruction quality of objects. The proposed mesh extraction method generates a smooth mesh model without holes, making it more suitable for the needs of shape perception through continuous touch. The constructed VOSRS can enable BVIs to learn and recognize the shape characteristics of virtual objects through force feedback during continuous touch. Our code can be available at https://github.com/CdpLab/VOSRS.git.