ENHANCING FRAUD DETECTION AND ANOMALY DETECTION IN RETAIL BANKING USING GENERATIVE AI AND MACHINE LEARNING MODELS

This study investigates the effectiveness of generative models and traditional classification models in detecting fraud and anomalies within the retail banking sector. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) were evaluated for their capability to generate realistic synthetic transaction data and identify anomalies, achieving anomaly detection accuracies of 91.2% and 93.5%, respectively. These models were also assessed using Inception Score and Fréchet Inception Distance (FID), with GANs exhibiting superior data realism. Among classification models, Gradient Boosting Machines (GBM) demonstrated the best performance, achieving an accuracy of 96.3%, a precision of 93.5%, a recall of 91.4%, and an AUC-ROC of 97.2%. Random Forest and Logistic Regression also performed well, though with slightly lower metrics.