Effective Performance Analysis of <scp>DCT OFDM</scp> ‐ <scp>IM</scp> Using Deep Learning Detector Under Different Fading Channels

ABSTRACT This work introduces an orthogonal frequency division multiplexing based discrete cosine transform assisted index modulation with novel signal identification technique. To take use of the design flexibility offered by the twice the number of accessible subcarriers under the same bandwidth, it combines the concepts of IM and DCT assisted Orthogonal Frequency Division Multiplexing (DCT‐OFDM). The performance of DCT‐OFDM‐IM in contrast to OFDM‐IM is enhanced in the proposed study by the employment of a deep learning detector. The Deep Learning based detector (DLD), in contrast to conventional detectors like Maximum Likelihood (ML), Greedy Detector (GD), Log Likelihood Ratio (LLR), and others, improves system performance and lowers system overhead. In order to perceive data bits at the OFDM‐IM system's receiver in Rayleigh, Rician, and Nakagami‐ m Fading channels, the proposed DLD uses a Deep Neural Network with completely automated linking layers. To start with, DLD is trained offline by assembling datasets of simulated results in order to enhance BER performance. Next, the model is trained to recognize DCT‐OFDM‐IM signals at the receiver under various fading channels. The results demonstrate that the DLD outperforms conventional approaches for all multipath fading channels in terms of BER, and that BER for DCT OFDM‐IM has improved over that of OFDM‐IM.