Reducing Sim-to-Real Domain Gaps of Visual Sensors for Learning Environment-Constrained Visuomotor Policy

Simulation engines enable safe training of robotic skills, but domain gaps between simulated and real sensors hinder deployment. However, existing pixel-level adaptation methods focus on the visual realism of generating images over task-specific learning, causing texture leakage and elimination. In this article, we introduce a learnable correlation-attentive and task-related generative adversarial network (LCTGAN), a novel unsupervised domain transfer network, with a correlative attention mechanism and a mask-level Q value mapping (MQM) consistency to enhance task awareness and bridge the domain gap of visual sensors in pixel-level perceptual manipulations. We also propose a Q-learning-based visuomotor policy to handle cluttered scenarios where objects may lack directly graspable configurations, which learns the synergies of three actions while considering environmental constraints. We further integrate LCTGAN into the learned policy to facilitate zero-shot sim-to-real policy transfer. Extensive experimental results validate the zero-shot sim-to-real generalization of our proposed visuomotor policy when deployed on a real robot. The supplementary video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/B6nODKkhzSw</uri>.