Deep Learning-Based Detection for RSMA With Orthogonal Time Frequency Space Modulation

The increasing demands for high-speed and reliable wireless communication in dynamic environments, such as high-Doppler scenarios, pose significant challenges to existing modulation and access schemes. Traditional techniques including orthogonal frequency division multiplexing (OFDM) and non-orthogonal multiple access (NOMA) are not suitable under such conditions due to their susceptibility to interference and Doppler effects. To address these challenges, this letter integrates rate-splitting multiple access (RSMA) with orthogonal time frequency space (OTFS) modulation, offering a novel robust solution for achieving high data rates and improved spectral efficiency. Further, we propose a novel dual block deep learning (DBDL) receiver for RSMA-OTFS, leveraging long short-term memory (LSTM) networks to enable signal detection without relying on successive interference cancellation. The DBDL receiver operates through two parallel deep neural networks, facilitating simultaneous decoding of common and private messages as a single process. We evaluate the symbol error rate (SER) performance of the DBDL receiver, demonstrating its parity with the optimal maximum likelihood (ML) receiver. Furthermore, we compare RSMA-OTFS with RSMA-OFDM, NOMA-OTFS, and NOMA-OFDM in terms of SER and rate region. Our results establish RSMA-OTFS as a promising solution due to its superior rate region performance across all other techniques, albeit at a slight expense in its SER compared to NOMA-OTFS.