Face-swapping Based Data Augmentation for ID Document and Selfie Face Verification

In this work, we focus on identity verification by matching selfies to identity (ID) documents depicting faces. When applied to this task, state-of-the-art deep face verification models tend to underperform due to lack of documents spe-cific visual features in the public datasets, and suffer from skin tone biases due to representation imbalances of skin tone groups in training data. To address both issues, we introduce a framework that generates ID documents con-taining synthetic face images to augment training datasets. Specifically, we transform selfies into document-style images by combining style banks of document templates and face-swapping generative models. Experiments on a public and a proprietary real-world dataset reveal that fine-tuning state-of-the-art face verification models with the proposed methodology yields 7.96% improvement in verification ac-curacy, while requiring only 25% of the original training data. Furthermore, improvements occur across all skin tone groups, including darker skin tones; though it is notori-ously hard to perform accurate verification for this group, we achieve more than 50% relative reduction in false acceptance and false rejection rate gaps w.r.t. lighter skin tones.