Controllable flow fields for high-fidelity virtual try-on with diffusion models

Abstract Virtual try-on is a challenging task that seeks to create realistic images of individuals wearing specific garments and has significant applications in e-commerce and virtual fashion. Existing approaches have made considerable advancements but come with large memory requirements and fail to preserve subtle garment details. To improve on these limitations, we present a new end-to-end diffusion-based framework that allows for more efficient and controllable detail generation. The key component of our approach is the COFFLOW module that extracts precise spatial correspondence from the target garment to the person image and incorporate this into the diffusion process as guidance for image synthesis. We also introduce a blending module that facilitates better visual quality in unmasked areas like face and limbs. We train our model freezing the pretrained diffusion U-Net and updating only the COFFLOW module, leading to a significantly lower cost during training. We evaluate our method on two standard benchmarks, VITON-HD and DressCode. Quantitatively and qualitatively, we show that our method significantly improves upon the prior diffusion-based methods, with better garment fitting, texture preservation, and realistic pose fitting, all while requiring lower computational resources.