Joint Antenna Position and Beamforming Optimization with Self-Interference Mitigation in Movable Antenna Aided ISAC System

Movable antennas (MAs) have shown significant potential in improving the performance of integrated sensing and communication (ISAC) systems. However, their application in integrated and cost-effective full-duplex (FD) monostatic systems remains underexplored. To bridge this research gap, we develop an MA-ISAC model within an FD monostatic framework, where the self-interference channel is modeled as a function of the antenna position vectors under the near-field channel condition. This model enables antenna position optimization for maximizing the weighted sum of communication capacity and sensing mutual information. The resulting optimization problem is non-convex making it challenging to solve optimally. To address this, we employ the fractional programming (FP) method and propose an alternating optimization (AO) algorithm that jointly optimizes the beamforming and antenna positions at the transceivers. Specifically, closed-form solutions for the transmit and receive beamforming matrices are derived using the Karush-Kuhn-Tucker (KKT) conditions, and a novel coarse-to-fine grained searching (CFGS) approach is used to determine high-quality sub-optimal antenna positions. Numerical results demonstrate that with strong self-interference cancellation (SIC) capabilities, MAs significantly enhance the overall performance and reliability of the ISAC system when utilizing our proposed algorithm, compared to conventional fixed-position antenna designs.