A year of machine learning for fraud prevention: a systematic meta-research study of 385 arxiv papers

We present a systematic meta-research study of machine-learning research on fraud prevention posted to arXiv from 1 August 2025 to 19 August 2026. Using 18 search queries, we screen 385 unique papers and identify 204 core contributions across payments, anti-money laundering, cryptocurrency crime, consumer scams, e-commerce abuse, and identity fraud. For 202 papers, we use full-text coding to distinguish methods that papers propose from those they merely run or mention, assess eight evaluation-quality properties and author affiliations, and record deployment evidence. Per-field Cohen’s κ ranges from 0.88 to 1.00 in a 40-paper reliability sample. We also verify every within-paper comparison with a conventional baseline and inspect every advertised code artifact with a resolvable location. The evidence supports one conclusion: evaluation discipline, rather than model capability, is the binding constraint in this literature. First, mention counts overstate practice. Graph neural networks are mentioned by 61% of papers but proposed by 17%; gradient-boosted trees are proposed by only 4%, yet a classical model is the strongest baseline in 35% of papers that run one. Among papers that propose and evaluate a model, 65% provide no extractable comparison with a conventional model. When such a comparison exists, proposed models win 77% of the time (46/60 papers, p<0.001), with median gains of +1.9 pp for PR-AUC and +5.0 pp for F1; we never pool metrics. Second, evaluation rarely reflects the problem the field claims to solve. Among papers running their own evaluation, 24% use an out-of-time split, 37% preserve realistic test-time prevalence, 26% report a cost- or budget-denominated result, and 44% address none of the three. These practices cluster strongly: 14 papers satisfy all three, compared with 4.0 expected under independence (p<0.0001). Third, the operational frontier is distinct from the academic literature. The 21 papers reporting production use or a pilot name incumbent systems, express results in review budgets and analyst time, and report latency—details largely absent elsewhere. Fourth, artifact availability falls far short of artifact claims. Although 27% of papers advertise an artifact, 29% of these artifacts are dead, empty, or have no resolvable location; only 11 papers document the environment, execution, and data access. Industry-involved papers provide more deployment evidence and use out-of-time evaluation more often, but use public data less often: the verifiability deficit lies in data, not code. Finally, apparent method trends are sensitive to the time window and disappear into intervals that include no change when a full year is analyzed. We identify ten research opportunities, including tabular foundation models for fraud, which appear in none of the 202 papers. All coding data, extracted comparisons, and audit scripts are released.