DL-NOMA-CancelNet: A Deep Learning Framework for Interference Cancellation Using CNN and BiLSTM

The rapid growth of user density and heterogeneous service requirements in 5G and emerging 6G networks has intensified multi-user interference, particularly in power-domain non-orthogonal multiple access (PD-NOMA) systems. Conventional successive interference cancellation (SIC) techniques suffer from error propagation and performance degradation under imperfect channel conditions, limiting their reliability in practical deployments. This paper proposes a deep learning–based interference cancellation framework that combines Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks for end-to-end signal detection in downlink NOMA systems. The proposed model exploits CNN layers to extract spatial interference features from complex-valued received signals, while BiLSTM layers capture bidirectional temporal dependencies caused by channel fading and symbol correlation. Unlike traditional SIC-based receivers, DL-NOMA-CancelNet directly reconstructs user signals without explicit channel estimation or manual interference subtraction. Simulation results obtained using MATLAB under Rayleigh fading and AWGN conditions demonstrate that our framework significantly outperforms conventional Least Squares (LS), Minimum Mean Square Error (MMSE), and single-network deep learning detectors. Specifically, this method achieves up to an 8–10× reduction in bit error rate (BER) across a wide signal-to-noise ratio (SNR) range, while maintaining low inference latency and moderate model complexity. These results confirm that our work provides a robust and scalable solution for interference mitigation in future 5G/6G downlink NOMA systems, particularly for ultra-reliable low-latency communications (URLLC) and massive machine-type communications (mMTC).