Diverse image inpainting by optimal transport

Deep generative models allow for photorealistic image inpainting. But for many applications, this is not enough: image inpainting also needs to be diverse. While several recent works investigate how to generate diverse image inpainting results, most of them have suffered a certain degree of mode collapse, and the diversity of generated results is limited. To address the challenge, we introduce a novel learning framework that can effectively simulate the uncertainty of image diversity for repair by optimal transport between the real image space and the conditional completion space. The proposed framework adopts a two-stage pipeline including pretext task stage and optimal transport task stage. For pretext task stage, we first pre-train an auto-encoder to model the embedding representation of training data by self-supervised learning, which learns the one-to-one mapping between the real image space and the latent variable space. In the optimal transport task stage, we take the latent variable space as a bridge (or common space), and utilize a distance-guided loss (maximizing the ratio of the distance between two completion images to the distance between its corresponding real images) and a contrastive learning loss to realize optimal transport between the real image space and the conditional completion spaces. The optimal transport learning could traverse the conditional completion spaces completely, which alleviates the mode collapses and diverse the image inpainting. For multiple datasets, such as CelebA-HQ, ImageNet, Places2, and Paris Street View, we demonstrate our approach could produce visually realistic and multiple solutions given a masked image