Transforming Simple Human Segmentation into Vivid Motion and Lifelike Images

Human appearance generation and pose manipulation have always been key topics in the field of image generation. Controlling the appearance and motions of generated images based on given conditions is a significant challenge. Through various information fusion methods, we can leverage simple segmentation and a reference image to control the appearance and pose of the generated image. Our approach SHS2Img aims to extract motion information using simple segmentation, such as human outlines, in situations where detailed segmentation is not available. Additionally, by integrating CLIP’s powerful semantic understanding capabilities, we can extract useful and relevant semantic information from the appearance reference image, providing our generation model with richer and more meaningful data. Once trained, our proposed method can be applied to tasks where only simple segmentation maps are available, enhancing the applicability of our work (usually, detailed segmentation maps like human parsing are time-consuming and expensive to obtain). Finally, leveraging the GAN architecture, we achieve highly efficient generation speed and high image quality, along with a high degree of controllability in altering the appearance and motions of individuals.