Hybrid Detection of Anomalies in Financial Transactions: A Rule-Based and Machine Learning Approach

This paper presents a hybrid methodology for detecting anomalies in financial transactions including card-initiated transactions and payments by combining rule-based logic with unsupervised machine learning techniques. Rule-based detection leverages expert-defined heuristics to flag transactions exhibiting high-risk behaviors such as card number, BIN, transaction amount, local time, date, expiry, MCC, country code, 3DSecurity Level, time interval, count, amount and location, plus excessive login attempts, abnormal transaction timing, and demographic inconsistencies. In parallel, three unsupervised models—Local Outlier Factor, One-Class SVM, and Autoencoder—are applied to extract structural and statistical anomalies without requiring labeled data. A weighted scoring mechanism aggregates model outputs to rank suspicious transactions, enhancing robustness through model complementarity. The methodology is evaluated on a synthetically enriched transactional dataset, demonstrating its ability to identify both interpretable and latent anomalies. Comparative results highlight the benefits of model diversity and reveal limited but meaningful overlap between rule-based and ML-based detections. The proposed framework offers transparency, flexibility, and practical scalability, making it well-suited for near real-time monitoring systems in the banking sector. Findings underscore the importance of multi-layered detection in modern anti-fraud card and payment management.