FraudShield-EAC: What a Synthetic East African Mobile-Money Fraud Benchmark Can and Cannot Show

A synthetic fraud benchmark is easy to build and hard to interpret: a high score may measure the generator rather than the model. We study this on FraudShield-EAC, a seeded, agent-based benchmark of East African digital payments (six channels, five countries, eight fraud scenarios) built to develop a fraud detection system. The benchmark is easy in a specific way: on the test period one engineered feature, a short-to-long-window velocity ratio, reaches an AUC of 0.879 on its own, so we report ranking metrics as margins over it. A calibrated XGBoost and LightGBM ensemble trained on all 792,162 training rows reaches an AUC of 0.975 (margin +0.096), but its recall at the specified flag threshold, 0.764, misses the target of 0.880, and more training rows did not close the gap in any run. At a fixed 30,000 rows, which rows were drawn moved recall at that threshold over a range of 0.107, against 0.032 for training on all rows; on AUC, volume mattered more. On a diagnostic model, ablations measure redundancy rather than importance (four disjoint feature groups each exceed AUC 0.80 alone), and leave-one-country-out and a novel variant held out in time are both null. The one variant specified in advance without burst structure, a reversal scam, is caught at 9.1% [4.2%, 18.4%] at a 1% false-positive budget, against 94.4% for the base scenarios. We distil the audit into a checklist for synthetic benchmarks. No fraud parameter is sourced, so every result describes the generator, not East African payments.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24150812
Primary Topic
Imbalanced Data Classification Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

FraudShield-EAC: What a Synthetic East African Mobile-Money Fraud Benchmark Can and Cannot Show

Marius Bayizere
Zenodo (CERN European Organization for Nuclear Research)
Imbalanced Data Classification Techniques
preprint

FraudShield-EAC: What a Synthetic East African Mobile-Money Fraud Benchmark Can and Cannot Show

Marius Bayizere
preprint en

Abstract

A synthetic fraud benchmark is easy to build and hard to interpret: a high score may measure the generator rather than the model. We study this on FraudShield-EAC, a seeded, agent-based benchmark of East African digital payments (six channels, five countries, eight fraud scenarios) built to develop a fraud detection system. The benchmark is easy in a specific way: on the test period one engineered feature, a short-to-long-window velocity ratio, reaches an AUC of 0.879 on its own, so we report ranking metrics as margins over it. A calibrated XGBoost and LightGBM ensemble trained on all 792,162 training rows reaches an AUC of 0.975 (margin +0.096), but its recall at the specified flag threshold, 0.764, misses the target of 0.880, and more training rows did not close the gap in any run. At a fixed 30,000 rows, which rows were drawn moved recall at that threshold over a range of 0.107, against 0.032 for training on all rows; on AUC, volume mattered more. On a diagnostic model, ablations measure redundancy rather than importance (four disjoint feature groups each exceed AUC 0.80 alone), and leave-one-country-out and a novel variant held out in time are both null. The one variant specified in advance without burst structure, a reversal scam, is caught at 9.1% [4.2%, 18.4%] at a 1% false-positive budget, against 94.4% for the base scenarios. We distil the audit into a checklist for synthetic benchmarks. No fraud parameter is sourced, so every result describes the generator, not East African payments.

Zenodo (CERN European Organization for Nuclear Research)
Imbalanced Data Classification Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon