Creative Design Transformations: From plain to Stylized Cloth Patterns Using SAM and Stable Diffusion Generative Model

Advances in image processing and artificial intelligence have brought about a substantial revolution in the design industry, particularly in image inpainting techniques, which have improved accuracy, originality, and creative patterns. Image inpainting restores blurred or corrupted areas of an image while enabling the creation of diverse visual patterns. This research focuses on transforming plain fabric designs into stylized cloth patterns using text-based prompts. It leverages a Vision Transformer- based pretrained Segment Anything Model (SAM) and the Stable Diffusion (SD) inpainting model to achieve precise, high-quality pattern generation. SAM ensures accurate object segmentation by precisely defining regions for modification, where SD concentrates on high-fidelity, photorealistic images with context-aware improvements and seamless transformations. By integrating these models (ViT-SAM-SD), visually appealing fabric patterns can be generated from text, offering greater control and innovation in design. The performance of the ViT-SAM-SD model is compared with MaskGiT, StyleGAN2, PSGAN, and VQGAN-based prompt-driven modelswith textile prompts. Experimental results indicate that the hybrid ViT-SAM-SD model achieves a low MSE of [Formula: see text], CLIP of [Formula: see text], and a PSNR of [Formula: see text], outperforming well compared with the four generative models. This approach simplifies and enhances fabric design, providing designers with a powerful tool for generating aesthetically striking patterns.