DSFace : Conditional Diffusion Inpainting for Sketch-to-Face Synthesis

Generating realistic human faces from monochromatic sketches is a challenging task with applications in forensic reconstruction, character design, and digital art. The limited semantic information in single-channel sketches makes this problem difficult, as they often lack fine details like expressions, skin tone and accessories. While GANs have shown promise, they suffer from unstable training and poor structural guidance. Diffusion models offer improved image generation, but struggle with monochrome inputs and high computational costs. To address these challenges, we propose DSFace, a latent diffusion-based framework that treats sketch-to-face generation as a conditional inpainting problem. DSFace utilises a frozen Paint-by-Example (PBE) inpainting diffusion model, conditioned with a ControlNet encoder, ensuring precise control over face synthesis. Our novel approach utilises a GAN-generated coarse image to compute DINO-V2 embeddings, which provide fine-grained details for improved facial and garment features. Trained on the CUFS dataset, DSFace achieves state-of-the-art performance, surpassing existing methods in visual realism, perceptual quality and structural alignment with the input sketches.