Simulation and Transfer of Physical Image Features Based on Recurrent Generative Adversarial Networks

Training machine learning models in virtual environments often fails to generalize effectively to real-world applications. The discrepancy between simulated and real environments results in suboptimal performance of the learned models. Various approaches have been proposed to address this “simulation-to-reality gap,” including refining training images within simulations or randomizing virtual environments to narrow the disparity. A CycleGAN model is used to convert simulated images into real ones for correction, but this method remains task-agnostic and does not preserve task-specific information, which can negatively affect task performance. Critical features, particularly those directly related to the task, may be lost during translation, which can compromise task success. To address this issue, conditional information is introduced during the image transformation process to ensure that task-relevant features, such as location data, are retained. This method makes the transformation task-aware rather than task-agnostic. Additionally, skip connections are implemented to mitigate blurring in the generated images. The proposed method has been trained on a custom dataset from a laboratory environment and validated on a robotic arm. This approach significantly improves the accuracy of image conversion tasks and enhances the efficiency of sim-to-real image translation.