Prompt Conditioned Batik Pattern Generation Using LoRA Weighted Diffusion Model With Classifier-Free Guidance

Batik, a significant element of Indonesian cultural heritage, is renowned for its intricate patterns and profound philosophical meanings. While preserving traditional batik is crucial, the creation of modern patterns is equally encouraged to keep the art form vibrant and evolving. Current research primarily focuses on batik classification, leaving a gap in the exploration of generative models for batik pattern creation. This paper investigates the application of text-to-image (T2I) generative models to synthesize batik motifs, leveraging latent diffusion models (LDM), Low-Rank Adaptation (LoRA), and classifier-free guidance. Our methodology employed a dataset of 20,000 batik images. Multimodal models such as LLaVA and BLIP were utilized to generate detailed captions for these images. A pretrained LDM was subsequently fine-tuned on its denoising U-Net part, either by naively fine-tuned the entire layer or by employing using LoRA. The fine-tuning process was critical in enhancing the model’s capability to generate high-quality and user-specific batik patterns. The results demonstrated that the LDM fine-tuned on the entire denoising U-Net with LLaVA-captioned images outperformed other models, achieving the lowest Fréchet Inception Distance (FID) and highest Inception Score (IS). The thoroughness of LLaVA captions proved superior to those generated by BLIP, emphasizing the significance of detailed image descriptions in generative tasks. Notably, the model not only replicated existing batik patterns but also innovatively combined multiple motifs and even able to create entirely new designs, as verified by batik expert. This research contributes to the field of computer-assisted batik pattern generation, providing significant advantages for batik artists, manufacturers, and users by accelerating the pattern creation process and expanding the possibilities of batik art.