Deep Reinforcement Learning for Anti-Jamming Dynamic Spectrum Access: A Bootstrap Ensemble Approach with Echo State Network and Idle-Ratio Change Detection

Dynamic spectrum access (DSA) is an effective technology to exploit spectrum for radio devices in complex electromagnetic environments. Systematic external jamming, such as swept and comb jamming, is a common form of jamming in anti-jamming communication scenarios. Deep reinforcement learning (DRL) is widely utilized to improve the performance of DSA. However, DRL-based DSA methods face challenges in generalizing across different jamming scenarios. In this paper, a DSA scheme based on a Bootstrap ensemble deep Q-network (BEDQN) integrated with an echo state network (ESN), termed ESN-BEDQN, is proposed to achieve fast and reliable access in scenarios where jamming patterns undergo sudden changes. The ESN provides low-complexity temporal memory to capture jamming patterns, while the BEDQN maintains multiple diverse readout heads to achieve fast exploration after ESN-BEDQN reset. Moreover, a jamming pattern change prediction method based on channel idle ratio detection using symmetric KL divergence is proposed to trigger network reset, i.e., Pred-Reset. Simulation results demonstrate that the ESN-BEDQN-based scheme achieves a near-zero collision rate under periodic jamming and recovers substantially faster than conventional deep Q-network (DQN)- and long short-term memory (LSTM)-DQN-based schemes in scenarios with abrupt jamming pattern changes. Furthermore, the Pred-Reset method can correctly capture jamming pattern changes and trigger network resets, achieving faster convergence than other baseline schemes across all tested scenarios.