FraudShield-EAC: What a Synthetic East African Mobile-Money Fraud Benchmark Can and Cannot Show
A synthetic fraud benchmark is easy to build and hard to interpret: a high score may measure the generator rather than the model. We study this on FraudShield-EAC, a seeded, agent-based benchmark of East African digital payments (six channels, five countries, eight fraud scenarios) built to develop a fraud detection system. The benchmark is easy in a specific way: on the test period one engineered feature, a short-to-long-window velocity ratio, reaches an AUC of 0.879 on its own, so we report ranking metrics as margins over it. A calibrated XGBoost and LightGBM ensemble trained on all 792,162 training rows reaches an AUC of 0.975 (margin +0.096), but its recall at the specified flag threshold, 0.764, misses the target of 0.880, and more training rows did not close the gap in any run. At a fixed 30,000 rows, which rows were drawn moved recall at that threshold over a range of 0.107, against 0.032 for training on all rows; on AUC, volume mattered more. On a diagnostic model, ablations measure redundancy rather than importance (four disjoint feature groups each exceed AUC 0.80 alone), and leave-one-country-out and a novel variant held out in time are both null. The one variant specified in advance without burst structure, a reversal scam, is caught at 9.1% [4.2%, 18.4%] at a 1% false-positive budget, against 94.4% for the base scenarios. We distil the audit into a checklist for synthetic benchmarks. No fraud parameter is sourced, so every result describes the generator, not East African payments.
Authors
- Marius Bayizere
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24150812
- Primary Topic
- Imbalanced Data Classification Techniques
- Type
- preprint