Structured light 3D imaging using digital twin and transfer learning

In recent years, deep learning techniques have been widely applied in computational optical 3D imaging. In fringe projection profilometry, high-precision phase information can be recovered from a single fringe image by training a deep neural network. However, training a neural network typically requires a significant amount of time and resources to collect training datasets. To address this issue, in this paper, we propose to establish a digital twin fringe projection system and enhance the virtual fringe patterns using domain randomization techniques. Virtual scanning is employed to generate many simulated fringe pattern images. Secondly, the pre-training of a U-Net neural network is conducted using the simulated fringe pattern images. Finally, transfer learning is introduced, and a small number of real fringe pattern images are used to fine-tune the parameters of the neural network. During the fine-tuning process, specifically for fringe analysis applications, this paper proposes and analyzes three U-Net neural network fine-tuning schemes: “from left to right”, “from top to bottom”, and global fine-tuning. Experimental results demonstrate that fine-tuning modules other than the “bottleneck” network module of the U-Net using the “from top to bottom” strategy yields the best transfer learning results, the neural network achieves a significant improvement in phase prediction accuracy. Compared to completely using a large amount of real data, the method described in this paper can achieve high-precision phase reconstruction results by training the neural network with only 20% of the real data.