Comparison of Machine Learning Methods for Fraudulent Transaction Classification

Financial fraud detection is a critical challenge faced by financial institutions, requiring robust machine learning techniques to distinguish fraudulent transactions (FTs) from legitimate ones. This study implements and compares supervised learning algorithms (XGBoost, Random Forest) with anomaly detection methods (Isolation Forest, Autoencoders) on a real-world dataset. Various preprocessing techniques, feature selection strategies, and performance metrics are evaluated to optimize fraud detection (FD) accuracy. The results demonstrate that XGBoost outperforms other models, achieving an F1-score of 0.89 and Precision-Recall (PR) AUC of 0.92, indicating its superior ability to handle class imbalance. While Random Forest (RF) also produces competitive results (F1-score of 0.85, PR AUC of 0.89), unsupervised methods such as Isolation Forest (IF) and Autoencoders (AE) exhibit significantly lower performance (F1-scores below 0.65). This study highlights the challenges of FD, including dataset limitations and the impact of imbalanced classes, and provides a benchmark for future research in financial fraud detection.