The Development and Validation of a Multi-Modal Fraud Detection Model for Second-Hand Transactions Leveraging Text and Images

With the rapid growth of second-hand trading platforms, fraudulent transactions have also increased significantly. However, existing fraud detection studies have primarily relied on structured or textual data, which limits their ability to capture the complex nature of fraud. This study proposes a multi-modal fraud detection model that integrates textual and visual information from second-hand trading posts collected from the Joonggonara platform. Text features were extracted using KLUE-BERT, while image features were obtained through the Swin Transformer. These embeddings were combined with structured data and input into an XGBoost classifier to predict fraudulent transactions. To address class imbalance, the SMOTE technique was applied to augment the minority (fraud) class. Experimental results demonstrate that the proposed multi-modal model, which combines structured, textual, and visual data, achieved the best overall performance compared to dual-modal models. These findings empirically confirm that integrating complementary information from text and images significantly enhances fraud detection capabilities. This study highlights the practical value of multi-modal approaches in second-hand transaction fraud detection and lays the groundwork for future research involving additional modalities and real-time detection systems.