CA-DIFF: enhancing stylistic consistency in handwriting generation—a diffusion model perspective

Handwritten text generation has garnered considerable attention due to its applications in data augmentation and personalized handwriting synthesis, supporting downstream tasks such as Handwritten Text Recognition (HTR) and writer identification. Traditional generative approaches, such as GAN-based models, often suffer from mode collapse, dependency on large training sets, and limited ability to capture global stylistic attributes. Transformer-based generative models, though powerful, are computationally intensive and struggle to generalize when trained on limited handwriting data. To overcome these limitations, we propose CA-DIFF (Cross-Attention Diffusion), a style-aware diffusion-based framework designed for generating high-fidelity and stylistically consistent handwritten word images. Our model incorporates a multi-stage cross-attention mechanism via a novel Style Encoder (SE) block that dynamically extracts writer-specific style features from intermediate layers of a MobileNet-based writer classifier. These features are integrated into the U-Net-based diffusion architecture at multiple stages (input, middle, and output), enabling fine-grained style preservation without relying on fixed writer IDs. Extensive experiments on benchmark handwriting datasets (IAM and CVL), along with evaluations on a low-resource historical handwriting dataset, demonstrate that our method improves downstream performance, achieving up to 6–10% higher writer classification accuracy over prior diffusion-based methods. These results validate CA-DIFF as a general and effective framework for historical handwritten text synthesis, adaptable to both large-scale and low-resource scenarios.