A spatiotemporal cell theory for cooperative pattern formation in reinforcement learning-driven evolutionary games

The emergence and stable evolution of cooperation among self-interested individuals is a central issue spanning evolutionary biology, social dynamics, and artificial intelligence. Conventional imitation-based evolutionary game models, lacking mechanisms for active exploration and experiential accumulation, often trap populations in suboptimal steady states and fail to explain the persistence of complex cooperative patterns. In this study, we construct a multi-agent reinforcement learning framework for the spatial snowdrift game and propose a spatiotemporal cell theory that systematically elucidates the mechanisms underlying cooperation driven by autonomous learning. Our results show that agents accumulating interaction experience via Q-learning achieve cooperation levels significantly surpassing classical replicator dynamics across a broad parameter range, with the system self-organizing into robust collective decision-making structures. From an experiential learning perspective, we reveal an endogenous mechanism of cooperative emergence, demonstrating that efficient cooperation can arise solely from individual exploration and local feedback, without external punishment, reputation mechanisms, or centralized control. The spatiotemporal cell theory provides a unified analytical framework that quantifies the coupling between microscopic learning trajectories and macroscopic pattern evolution. Based on this theory, we derive a contour plot of the fraction of cooperators in the α-γ parameter plane that delineates cooperative stability, breakdown, and frozen defect-line phases, and uncover two distinct evolution pathways: cooperative amplification induced by synchronous exploration and noise accumulation driven by asynchronous exploration. This work deepens the understanding of cooperative evolution and provides theoretical support for designing decentralized adaptive multi-agent systems.