Optimal Beamforming for Covert Communication in MIMO Relay Systems

ABSTRACT This paper investigates the optimal beamforming design for covert communication in a multiple‐input multiple‐output (MIMO) relay system consisting of source, destination, relay and warden. For both scenarios when each transmitting node knows the perfect or partial warden's channel state information (CSI), we first develop theoretical models to depict the inherent relationship between the beamforming matrices at source/relay and end‐to‐end covert performance of the system regard the minimum detection error probability and covert capacity. Based on these models, we then formulate the optimal beamforming for covert capacity maximization under perfect and partial CSI as linear non‐convex (LNC) and non‐linear non‐convex (NLNC) optimization problems, respectively. We then apply Lagrangian and Hadamard inequality methods to solve the LNC optimization problem to determine the closed‐form expressions for the optimal beamforming matrices at source/relay, based on which a stochastic gradient descent algorithm is developed to evaluate the maximum covert capacity. For the complicated NLNC problem, we propose an efficient search algorithm based on S‐procedure lemma and interior point methods to identify the optimal beamforming matrices to reach the maximum covert capacity. Finally, we provide extensive numerical results to illustrate the covert performance enhancement from adopting the optimal beamforming in MIMO relay systems.