Variational Autoencoders for Detecting Anomalous and Fraudulent Transactions in Financial Systems

The rapid expansion of digital financial transactions is leading to an increase in both the complexity and sophistication of fraudulent schemes, many of which circumvent traditional classification systems based on rules and supervised learning. In this paper, we investigate the effectiveness of variational autoencoders (VAE) as an unsupervised method for detecting both anomalous and fraudulent financial transactions. We compare three approaches: classical supervised models (logistic regression, XGBoost), the VAE trained exclusively on legitimate transactions to detect outliers due to reconstruction error, and a hybrid model that combines VAE-based anomaly detection with a classifier for fraud labeling. Furthermore, based on the VAEs, we try to generate synthetic fraudulent transactions that simulate new fraud models designed to deceive standard models. Our results demonstrate that the VAEs are effective in detecting previously unknown fraudulent behavior and offer increased flexibility and explanatory power through their latent space. This research highlights the potential of deep generative models to complement traditional financial fraud detection systems and provides a foundation for anomaly-aware hybrid architectures in the financial domain.