Joint Beamforming and Reflector Design for Near-Field RIS-Assisted MISO Systems via DDPG Approach

Reconfigurable Intelligent Surface (RIS) devices inherently lack A/D and D/A converters and power amplifiers, resulting in limited signal processing capability and effective radiation range. The design of RIS reflectors typically requires channel state information (CSI). Conventional RIS designs for far-field transmission simplify the process by considering only the direction of radiation. However, in near-field RIS-assisted mmWave transmissions, the near-field effect necessitates consideration of both the direction of propagation and the distance between the transmitter and receiver in the near-field model. Consequently, complete CSI is essential for accurately configuring reflectors for near-field propagation. However, acquiring complete CSI is impractical, especially in mmWave RIS-assisted massive MIMO systems. In this paper, we propose a method for jointly optimizing beamforming and RIS reflectors in RIS-assisted multiple-input single-output (MISO) systems using Deep Deterministic Policy Gradient (DDPG) reinforcement learning (RL) with and without CSI. When near-field CSI is unavailable, the proposed design directly configures the reflectors by leveraging the received signal power corresponding to pilot signals. Alternatively, with near-field CSI, the proposed joint design can explore a more feasible solution set through the DDPG framework, achieving superior performance. Simulation results provide valuable insights into beamforming and RIS design in near-field channels, demonstrating that the proposed methods outperform existing approaches. Furthermore, the proposed beamforming and RIS design without CSI eliminates the need for explicit channel estimation, significantly enhancing its practicality for real-world system applications.