BI-GBDT: A Graph-Free Behavioral Interaction-Aware Gradient Boosting Framework for Fraud Detection in Large-Scale Payment Systems

Detecting fraudulent and anomalous transactions in large-scale digital payment systems is significantly challenging due to severe class imbalance and the fact that transactional risk is tightly coupled to the historical interactions and behaviors of transacting parties. In this study, a scalable Behavioral Interaction-Aware Gradient Boosting (BI-GBDT) framework is proposed for anomaly detection in tabular transaction data to overcome these challenges. The methodology models sending and receiving behaviors separately through direction-specific clustering based on transaction frequency and amount. Each transaction is characterized by cluster-pair prevalence ratios, which capture the population-level prevalence of sender–receiver interaction patterns. To handle extreme class imbalance, all transactions are clustered, and a cluster-level risk score is computed as the ratio of anomalous transactions to the total number of transactions within each cluster. This score is incorporated as a feature, serving as a behavioral risk prior highlighting concentrated anomaly. These interaction-aware features are integrated into a GBDT in a big data environment. Experiments were conducted on a large masked real-world payment dataset spanning six months and containing more than 456 million transactions, with the prediction task defined as binary classification between fraudulent and non-fraudulent transactions. Unlike standard GBDT models trained only on transactional attributes and graph-based approaches, BI-GBDT captures sender–receiver interaction patterns in a graph-free manner and outperforms a baseline GBDT, reducing the false positive rate from 37.0% to 4.3%, increasing recall from 52.3% to 72.0%, and improving accuracy from 63.0% to 95.7%.