Tackling Class Imbalance in Financial Fraud Detection with Relational Graph Transformers and Contrastive Learning

Detecting fraud on financial transaction networks is difficult for three reasons that tend to appear together: fraudulent users deliberately disguise their behavior, their activity is linked to other entities through several distinct relation types, such as a shared merchant, a shared rating or a short time window, and they make up only a small fraction of the accounts in an otherwise benign population. Standard graph neural networks (GNNs) fold these relational cues into a single homogeneous neighbor aggregation and supervise every node with a label-balanced loss, which caps both their expressiveness and their recall on the rare fraud class. We introduce RGT-Fraud, a relational graph transformer that couples relation-aware message passing with global self-attention through a per-node topology gate, and we train it with an imbalance-aware contrastive (IAC) objective. The model operates on a multi-relation graph whose labeled nodes are the review or account entities to be classified, connected by three complementary relation types that encode a shared merchant, a shared rating or amount, and a short co-occurrence time window; merchants, ratings and timestamps are transaction attributes that define these relations rather than independently featured nodes. This is the standard construction of the YelpChi, Amazon and T-Finance benchmarks, and the same formulation extends to additional node types when a dataset exposes them. A relational position encoding writes edge-type information into the attention scores, and a topology-gated block lets each node decide how much local aggregation and how much long-range attention to use. The IAC objective adds three terms: a focal cross-entropy term for class imbalance, a supervised contrastive term that tightens the fraud cluster in embedding space, and a neighbor-consistency term that damps adversarial structural noise. On three public benchmarks (YelpChi, Amazon and T-Finance) RGT-Fraud improves F1-macro by 1.4 to 1.6 points and AUC by 0.9 to 1.5 points over the strongest competitor among twelve baselines, which include the recent heterogeneous multi-relational detector TMR-GGNN, and it keeps more than 70% F1-macro under 30% label noise. Ablations attribute a clear, positive contribution to each module and each loss term. Taken together, the framework offers an interpretable semantic layer that links raw transaction streams, a graph inference engine and human risk analysts inside financial expert systems.