Achieve Sim2Real based on semantic constrained cycle generative adversarial network

In the field of vision-based control systems, the discrepancy between simulator and real-world environments renders models trained in simulators ineffective in real-world scenarios. Previous approaches have attempted to mitigate this issue by mapping the simulator and real-world into a shared latent space, but this can result in the loss of semantic information relevant to decision-making in the images. In this paper, we propose a method called semantically constrained CycleGAN (SCCGAN) to address these limitations. SCCGAN extracts semantic information from generated images and compares it with the original images to ensure consistency. Experimental results demonstrate that our method preserves the semantic information of the original images during the generation process, enabling the transfer of decision models from simulators to the real world. By leveraging semantic constraints, SCCGAN facilitates the effective migration of decision models, bridging the gap between simulated and real-world environments in vision-based control systems.