TriMAE: Fashion Visual Search with Triplet Masked Auto Encoder Vision Transformer

Visual search is a technology that identifies images similar to a provided query image and presents results ranked by similarity. In the realm of apparel, this innovative tool revolutionizes shopping by enabling users to find desired items based on visual preference effortlessly. Visual search in the fashion domain poses significant challenges due to variations in minute details, multiple garments in a single image, discrepancies between user-taken and catalog images, and the inherent flexibility of clothing. To tackle these challenges, we introduce an innovative method that combines Triplet Neural Networks with a Masked Autoencoder Vision Transformer (MAE-ViT) model to enhance visual search tasks. Our method capitalizes on the strengths of both architectures to extract robust features and enhance the understanding of similarities and differences between images. Specifically, we employ a triplet loss function during network training to enhance the understanding of image similarity. We validate our approach using the DeepFashion In-shop dataset, encompassing a wide range of clothing image categories. Through extensive experiments on this benchmark, our model achieves an impressive Recall@1 of 93.2 % for visual search.