Signal detection of massive MIMO-OTFS using DNN algorithm with diverse channel state estimation

Massive multiple-input multiple-output orthogonal time frequency space (M-MIMO-OTFS) is a promising waveform for beyond 5G (B5G) systems, offering high spectral efficiency and robustness in time-varying channels. However, signal detection is challenged by high-dimensional processing and severe intersymbol interference under Rayleigh fading. A Modified Deep Neural Network (M-DNN) with prior channel state information (CSI) estimation was employed to enhance detection accuracy in large-scale M-MIMO-OTFS systems. Simulation results show SNR reductions of 2.9 dB, 5.8 dB, and 7.2 dB at a BER of 10 -5 for 64×64, 256×256, and 512×512 configurations, respectively, demonstrating improved detection with higher MIMO dimensions. For a 512×512 system with a 10% CSI error variance, the performance degrades slightly to 9.9 dB, indicating robustness to CSI imperfections. Power spectral density (PSD) analysis revealed a −110 dB improvement over conventional methods, enhancing the spectral efficiency and reducing out-of-band emissions. The combination of deep-learning-based detection and CSI estimation supports reliable BER performance and optimized spectral usage, making the approach suitable for large-scale M-MIMO-OTFS deployment in B5G and 6G networks.