Dual Alignment-enhanced Fashion Vision-Language Pre-training

Fashion vision-language pre-training (VLP) models have demonstrated remarkable capabilities in excelling at a wide range of fashion cross-modal tasks. However, current models still face three notable limitations: 1) an inability to discern varying levels of consistency between textual descriptions and multi-view images, 2) a deficiency in explicit fine-grained alignment between images and text, and 3) a lack of specific supervision mechanisms for facilitating global joint-embedding learning. To address these limitations, we propose a novel dual alignment-enhanced fashion VLP model. This model delves deeply into the rich resources of multi-view images and semantic attributes associated with each fashion item. Notably, we introduce two novel pretraining tasks: Multi-grained Adaptive Image-Text Alignment (MAITA) and Joint Embedding-oriented Alignment (JEA). MAITA focuses on optimizing the text/image encoder by orchestrating adaptive alignment between multi-view images and input text. This encompasses both coarse-grained and fine-grained alignment strategies to enrich semantic understanding, while JEA is devised to supervise the fine-grained semantic learning process of the global multimodal joint embedding. Experimental results spanning four diverse downstream tasks, including cross-modal retrieval, text-guided image retrieval, category recognition, and subcategory recognition, substantiate the significant performance superiority of our model over prior state-of-the-art fashion VLP models.