Diffusedesigner: sketch-based controllable clothing image generation

Abstract A tech pack is a set of documents that guides the production of a fashion product and includes all the necessary details to create the garment as designed. It helps to communicate between designers and manufacturers by specifying the detailed requirements for actual garment production. The motivation for this study originates from the resemblance between the flat sketch included in the tech pack and the final garment design. We propose DiffuseDesigner , a sketch-based controllable clothing image generation method, which is trained using pseudo flat sketches and prompts describing the desired output design. In light of this, we collected a large-scale fashion image dataset composed of multiple categories of clothing from three shopping websites. An edge detection algorithm was applied for each image to generate a pseudo flat sketch, and textual prompt information was extracted using a CLIP. Then, we constructed a triplet consisting of the pseudo flat sketch, textual prompt, and original image, and fine-tuned ControlNet to synthesize clothing images based on a pseudo flat sketch while leveraging the prompt information to guide the generation process. We conducted extensive experiments for performance evaluation based on the prompt the user wishes to generate, and quantitatively verified the effectiveness of clothing generation using five evaluation metrics. Finally, through variations in the prompt and pseudo flat sketch, we visually confirmed that the trained model was well-controlled in the direction desired by the designer.