Deep spiking Q-networks for turn-based game environments: encoding choices and energy trade-offs

Abstract Deep Reinforcement Learning (DRL) has achieved remarkable success across various domains but at the cost of substantial energy consumption. Spiking Neural Networks (SNNs), inspired by biological neural processing, offer a promising solution through event-driven, sparse computations. To evaluate the capability of SNNs in discrete, strategic decision-making settings, we adopt the Deep Spiking Q-Network (DSQN) framework, whose viability has been established in foundational studies. However, these prior works predominantly restricted DSQN’s application to reactive control tasks and high-frequency visual environments such as Atari games, leaving its efficacy in deliberate, long-horizon strategic reasoning and discrete turn-based environments unexplored. This paper addresses this gap by evaluating the DSQN architecture against traditional Deep Q-Networks (DQN) to investigate whether spike-based value estimation can achieve comparable performance in turn-based strategic tasks. To enable this comparison, we adopt two well-established open-source turn-based environments: Tic-Tac-Toe as a baseline micro-benchmark for turn-based structures, and Connect 4 to assess architectural scalability under complex game environments. Within this framework, we then systematically analyze six distinct SNN input encoding methods: Population, Count Rate, Time-to-First-Spike (TTFS), Rank Order Coding (ROC), Sparse Distributed Representation (SDR), and Burst encoding. Our results demonstrate that DSQNs achieve up to 82.7% and 80.5% detailed analytical energy savings in Tic-Tac-Toe and Connect 4 respectively compared to traditional DQNs, while maintaining comparable strategic performance. In complex environments, a clear trade-off emerges: Population encoding provides the highest strategic success rate (92.80% ± 2.28% win+draw in Connect 4), TTFS offers the maximum energy efficiency at the cost of a reduced win+draw rate (84.80% ± 1.30%), and ROC emerges as an optimal balanced choice, preserving a competitive win+draw rate of 91.80% ± 3.03 while delivering 80.1% detailed energy savings in Connect 4. These findings offer practical insights into selecting SNN encoding strategies tailored to accuracy, and energy constraints.