Classification of fashion e-commerce products using ResNet-BERT multi-modal deep learning and transfer learning optimization

As the fashion e-commerce markets rapidly develop, tens of thousands of products are registered daily on e-commerce platforms. Individual sellers register products after setting up a product category directly on a fashion e-commerce platform. However, many sellers fail to find a suitable category and mistakenly register their products under incorrect ones. Precise category matching is important for increasing sales through search optimization and accurate product exposure. However, manually correcting registered categories is time-consuming and costly for platform managers. To resolve this problem, this study proposes a methodology for fashion e-commerce product classification based on multi-modal deep learning and transfer learning. Through the proposed methodology, three challenges in classifying fashion e-commerce products are addressed. First, the issue of extremely biased e-commerce data is addressed through under-sampling. Second, multi-modal deep learning enables the model to simultaneously use input data in different formats, which helps mitigate the impact of noisy and low-quality e-commerce data by providing richer information.Finally, the high computational cost and long training times involved in training deep learning models with both image and text data are mitigated by leveraging transfer learning. In this study, three strategies for transfer learning to fine-tune the image and text modules are presented. In addition, five methods for fusing feature vectors extracted from a single modal into one and six strategies for fine-tuning multi-modal models are presented, featuring a total of 14 strategies. The study shows that multi-modal models outperform unimodal models based solely on text or image. It also suggests the optimal conditions for classifying e-commerce products, helping fashion e-commerce practitioners construct models tailored to their respective business environments more efficiently.