Synthesis of Comic-Style Portraits Using Combination of CycleGAN and Pix2Pix

This paper proposes a system for converting face photos into portraits with specific charcoal sketch-like style. NPR (non-photorealistic rendering) for converting real photo images into anime-style images have been studied. Its promising application is the creation of users’ portraits on social networks for preventing leaks of personal information. An image-to-image transformation using GAN (Generative Adversarial Network), such as CycleGAN or Pix2Pix, is expected to be used for this goal. However, it is difficult to generate portraits with specific comic styles that satisfy two conditions at the same time: preserving photorealistic features and reproducing the comic styles. For example, replacing too many photorealistic features on the face with comic style will destroy users’ identity, such as eyes shape. To solve this problem, the proposed system combines CycleGAN and Pix2Pix. CycleGAN is used to generate paired examples for Pix2Pix and Pix2Pix learns photo-to-comic transformation. To further emphasize a comic style, this paper also proposes two extensions, which divides face images into several parts such as hair and eye parts. The quality of generated images are evaluated with questionnaire by 50 answerers. The results show that the images generated by the proposed system are highly evaluated than the images generated by CycleGAN in terms of preserving the features in a photo and reproducing a comic style. Although the segmentation tend to collapse the generated images, it can reproduce a comic style effectively, especially on facial parts.