Credit card fraud detection using BERT based transformer architecture

With the rapid growth of online shopping, credit cards have become one of the most widely used payment methods. However, this increased usage has also led to a rise in fraudulent transactions, resulting in significant financial losses for both customers and merchants. Over the years, several machine learning, deep learning, and graph neural network-based approaches have been proposed for credit card fraud detection. However, these models often fail to achieve satisfactory detection performance, largely due to the highly imbalanced nature of credit card datasets. More recently, with the advent of transformer-based language models, researchers have begun exploring their potential for tabular data prediction tasks. In this work, we demonstrate the effectiveness of DistilBERT, a lightweight variant of the BERT transformer model, for credit card fraud detection on tabular datasets. We present two different approaches, namely Retrieval-Augmented Classification (RAC) and continued pre-training followed by full fine-tuning. These approaches are evaluated using different input text representations, feature sets, and sampling techniques. To address the class imbalance problem, we employ two widely used sampling techniques: Random Under-Sampling (RUS) and the Synthetic Minority Over-sampling Technique (SMOTE). Experimental results demonstrate that the proposed fine-tuning approaches outperform the considered baseline methods on two credit card fraud detection datasets.