Self-supervised Outlier Segmentation-guided Face De-occlusion Using 3D Facial Reconstruction

Occlusion is a common occurrence in facial images and videos captured in the wild. Effective face de-occlusion is of paramount importance for enhancing the robustness of user identity verification and security systems. To this end, we propose a novel self-supervised GAN-based framework for face de-occlusion in this study, which integrates 3D facial reconstruction with outlier segmentation guidance, termed as 3DOG-GAN. In 3DOG-GAN, we introduce an outlier segmentation module that utilizes statistical priors to generate accurate occlusion masks, facilitating the de-occlusion process. Furthermore, we design a GAN-based dual-branch module, which is capable of simultaneously generating the occlusion mask and the de-occluded face. Extensive experiments on the widely-used datasets demonstrate the superior performance of 3DOG-GAN on existing methods.