A hybrid CNN-LSTM-transformer model for window-based anomalous transaction detection in financial data

Some of the challenges of financial transaction anomaly detection include the fact that fraud is rare, adaptive and costly to operate. In this study, we propose the novel stacked convolutional neural network-long short-term memory-Transformer (CNN-LSTM-Transformer) model to detect anomalous transactions in a window-based approach and empirically analyze it with the public Credit Card Fraud Detection Dataset 2023. The dataset contains 568,630 anonymized records with 28 principal component analysis (PCA)-transformed features and transaction amount. To provide a conservative assessment, controlled imbalance experiments were conducted at transaction-level fraud prevalence of 10%, 5%, 3%, and 1%. Fraud-if-any labeling was adopted for window-level classification because majority voting suppresses rare fraud events in low-prevalence settings. Under the 1% transaction-level fraud setting with window size 16, and averaged over 10 random seeds, the CNN-LSTM-Transformer achieved 97.07% accuracy, 94.42% precision, 85.70% recall, 89.84% F1-score, 95.32% area under the receiver operating characteristic curve (ROC-AUC), and 90.05% area under the precision-recall curve (PR-AUC). Fair baseline comparisons were conducted by training Logistic Regression, Random Forest, and XGBoost on the same flattened window representation. The CNN-LSTM-Transformer performed on par with XGBoost, with no statistically significant difference in F1-score after multiple-comparison adjustment, while both clearly outperformed Logistic Regression and Random Forest; XGBoost retained a small advantage in PR-AUC and precision, and the CNN-LSTM-Transformer achieved slightly higher recall. The results indicate that gradient-boosted trees remain highly effective on PCA-transformed tabular fraud benchmarks, whereas the neural architecture should be interpreted as a window-based representation-learning approach rather than as proof of deployment-ready chronological behaviour modelling.