A Diffusion-Based Approach for Text-to-Face Image Generation Using Multi-Granularity Features

Text-to-Face generation aimed at generating realistic face images based on natural language descriptions and ensure the semantic consistency between the text and the images. This research presents a novel model, MGF-DiT, based on the Diffusion Transformer (DiT), aimed at addressing the issue of semantic inconsistency between text descriptions and generated images in text-to-face generation tasks. The model introduces two key components: a Text-Driven Mapping Module (TDM) and an Attribute Words and Text Fusion Guidance Module (AWTF). MGF-DiT follows a two-stage approach. In the first stage, TDM aligns semantic information by mapping random samples into the image latent space using text vectors. In the second stage, AWTF applies multi-granularity constraints through text vectors and attribute word vectors to further enhance semantic consistency. Experimental results demonstrate that, compared with the baseline model DiT, the FID score, IS score and RP value of this model on the FFHQ-Text dataset are increased by 3.02%, 10.67% and 12.81% respectively; on the Face2Text dataset, the FID score, IS score and RP value are increased by 23.4%, 7.93% and 11.27% respectively. From the above experimental results, MGF-DiT significantly improves semantic alignment between text and generated images, while also enhancing image quality compared to baseline models.