MAC-VTON: Multi-modal Attention Conditioning for Virtual Try-on with Diffusion-Based Inpainting

Virtual try-on has emerged as a prominent field of research because of its wide application in ecommerce. The key challenge in this task is producing realistic try-on images, where the garment is warped to accurately fit the person's body. The try-on generation quality is of significance to provide desirable and realistic results. In this work, we improve the realism and accuracy of virtual try-on by enhancing the architecture to accept attention conditioning. Instead of using multiple ControlNets, we introduce a novel Multi Zero Cross Residual Attention Block (MZB) that replaces the standard attention mechanism, enabling the simultaneous incorporation of multiple conditioning inputs. These include DinoV2 embeddings for rich semantic cloth representation, color histograms to preserve garment color fidelity, and skin maps to ensure coherence between the garment and exposed skin areas. This design allows the diffusion model to effectively handle complex garment-body interactions with greater precision. Through extensive experiments, our method demonstrates superior control over garment alignment and visual realism, achieving competitive results on standard virtual try-on benchmarks.