On the Rate Region of the Downlink NOMA System With Improper Signaling and Imperfect SIC

Non-orthogonal multiple access (NOMA) is a promising technology garnering significant attention among the Internet of Things (IoT) community due to its superior spectral efficiency. This work addresses the rate region boundary enhancement of downlink NOMA systems under imperfect successive interference cancellation (SIC) with advanced improper Gaussian signaling (IGS), which provides additional degrees of freedom for system design. We investigate a universal scenario in which two users adopt improper signaling and their transmit powers are optimized. We first formulate the achievable rate of both users in terms of the impropriety degree of the IGS. First, the analytical expressions for the best improper transmission are characterized by jointly optimizing the users’ power and the impropriety degree for the perfect SIC case. Then, a deep Q network (DQN)-based approach is provided to find the rate region of the IGS-aided NOMA system under imperfect SIC. Simulations presented for the downlink NOMA system support the analysis, illustrating that IGS can efficiently enhance the rate region of the NOMA system compared to proper signaling.