Mathematical Framework for ABM-MARL Integration in Financial Systems: A Discrete Multi-Agent Population-Strategy Game Approach

Modern financial markets feature complex interactions between vast populations of rule- based agents and adaptive algorithmic traders, yet existing models treat these layers separately. We introduce a discrete-time population-strategy game unifying Agent-Based Modeling (ABM) and Multi-Agent Reinforcement Learning (MARL). The framework’s core innovations are: (1) an asymmetric bilevel architecture where strategic agents optimize over population distributions while endogenously shaping them; (2) a rigorous treatment of system noise via martingale difference processes with bounded moments; and (3) a verifiable, post-convergence spectral stability certificate (ρ(∇µΦ) < 1). Under mild conditions, we prove: • Existence of a Mean-Field Equilibrium • Almost sure convergence of a two-timescale learning dynamic (policy gradient + population dynamics) to this equilibrium • An O(N−1/2) finite-population approximation error Our linearly scalable Population-Strategy Policy Gradient (PSPG) algorithm enables tractable computation. We apply the framework to a synthetic market-making environment. Experiments demonstrate emergent critical thresholds (e.g., an 8.2 bps spread bifurcating stable and fragmented regimes in our calibrated model) and a 23% volatility reduction versus a pure ABM system. This framework bridges ABM’s emergent heterogeneity with MARL’s strategic adaptation, addressing key gaps in mean-field game theory and enabling a path toward real-world deployment with verifiable stability.