Collaborative Hierarchical Multi-Agent Learning for RIS-Assisted Multi-Cell Integrated Sensing and Communication

The convergence of integrated sensing and communication (ISAC) and reconfigurable intelligent surfaces (RISs) offers new opportunities for building adaptive and energy-efficient 6G wireless networks. By jointly exploiting sensing information and programmable wireless propagation, these technologies can support intelligent network operation in highly dynamic environments. However, realizing these benefits in multi-user and multi-cell networks requires jointly controlling several strongly interconnected variables, including transmit beamforming, RIS configurations, power allocation, and user association. This problem becomes increasingly difficult in the presence of user mobility, imperfect channel state information (CSI), and strong interference among neighboring cells. Existing iterative optimization algorithms can become computationally demanding for large-scale networks, whereas conventional flat multi-agent reinforcement learning (MARL) approaches often struggle with the rapidly growing action space and limited inter-agent coordination. To address these limitations, this study develops an HDRL-ISAC framework based on hierarchical multi-agent learning and centralized training with decentralized execution (CTDE). The proposed framework assigns different decision-making roles across the hierarchy, allowing high-level agents to coordinate network-wide cellular decisions while lower-level agents perform user-specific resource optimization. This hierarchical decomposition improves the learning efficiency and enables scalable coordination in multi-cell environments. Furthermore, sensing information is incorporated into the learning process through EKF-based mobility prediction, enabling agents to anticipate user movement and adapt their resource allocation decisions accordingly. Lightweight information exchange among neighboring cells is also introduced to facilitate interference-aware coordination under imperfect CSI. Simulation results demonstrate that the proposed HDRL-ISAC approach improves energy efficiency by up to 35%, reduces inter-cell interference leakage by up to 6 dB, and decreases ping-pong handovers by up to 40% compared to benchmark schemes. Crucially, the proposed architecture achieves approximately 1.8× faster training convergence than conventional flat MARL approaches while avoiding the prohibitive online computational delays associated with conventional iterative optimization methods. These findings indicate that hierarchical and sensing-aware multi-agent learning provides a promising approach for scalable resource management in future RIS-assisted multi-cell ISAC networks.