Antenna Selection in MIMO–NOMA Systems: A New Approach for Physical Layer Security Enhancement

ABSTRACT Non‐Orthogonal Multiple Access (NOMA) is known as a promising technology for next‐generation wireless communication networks. In this paper, three low‐complexity antenna selection schemes are proposed that aim to enhance the physical layer security (PLS) of a multiple‐input multiple‐output (MIMO) NOMA system. The system is composed of a single transmitter, multiple legitimate users, and one eavesdropper, all equipped with multiple antennas. The first scheme maximizes the secrecy sum‐rate (SSR) of the system when the eavesdropper's channel state information (CSI) is known, whereas the second scheme maximizes SSR in the absence of CSI. In the third scheme, the emphasis shifts towards fairness, aiming to maximize the minimum secrecy rate across all users. Numerical results demonstrate that the performance of the first two proposed schemes is very close to that of the optimal case, albeit with much lower computational complexity. It is also observed that the third scheme enhances fairness by balancing users' secrecy rates, though with a moderate decrease in the overall secrecy sum‐rate.