Codebook-Based secure sum-Rate optimization via multi-Head DQN in STAR-RIS-Aided mmwave networks

We address secrecy sum-rate (SSR) maximization in beyond-5G millimeter-wave downlinks assisted by Simultaneously Transmitting and Reflecting Reconfigurable Intelligent Surfaces (STAR-RIS) with codebook-based beamforming at both the base station and the surface. The system operates under a 3GPP-compliant wideband OFDM framework that captures beam squint, inter-user interference, and per-tone power constraints, while explicitly modeling passive eavesdroppers under channel and location uncertainty. To handle the large discrete design space of BS/STAR-RIS codebook selections, artificial-noise (AN) directions, and power allocation, we propose a factorized multi-head deep Q-network (MH-DQN) that decomposes the joint optimization into coordinated sub-tasks with separate network heads. This factorization preserves compatibility with hardware-constrained codebooks and enables scalable learning over large combinatorial action spaces. Numerical results over standardized 3GPP CDL indoor and outdoor channels show that MH-DQN improves SSR by 13.3% over Greedy codebook search and by 31.0% over Proximal Policy Optimization (PPO) across a wide SNR range. Multi-seed experiments indicate stable convergence, and Conditional Value-at-Risk (CVaR) analysis suggests improved worst-case secrecy performance under eavesdropper uncertainty. These findings indicate that codebook-driven, factorized deep reinforcement learning is a practical and scalable approach for secure beamforming in STAR-RIS-assisted wideband mmWave networks.