Unsupervised highlight removal with an enhanced CycleGAN and differentiable renderer

The primary objective of this study is to remove highlights from images of mirror-like objects. Currently, highlight removal models in the field of computer vision are categorized into unsupervised and supervised models. Obtaining paired datasets with and without highlights is a key technique for supervised learning. In this study, a large-scale paired synthetic image dataset is generated using a differentiable neural renderer and specular reflection methods, with all camera parameters known. An enhanced unsupervised CycleGAN network is used, which separates images into foreground and background with geometric consistency constraints, applying highlight removal only to the foreground. This approach overcomes the limitations of traditional dual adversarial models. Finally, images are synthesized using a differentiable renderer to iteratively train the CycleGAN network model. Our method outperforms several advanced methods upon comparison. The code is available at https://github.com/lixian-wtu/HighlightRemoval.