Convolutional Neural Network-Based Channel Estimation for mmWave MIMO-OTFS Systems

The amalgamation of millimeter-wave (mmWave) communications and multiple-input multiple-output (MIMO) orthogonal time frequency space (OTFS) systems holds significant promise for next-generation wireless networks, offering high data rates and robust performance in high-mobility environments. This paper presents a novel convolutional neural network (CNN)-based channel estimation framework that unfolds the sparse Bayesian learning (SBL) algorithm into a deep neural network (DNN) for mmWave MIMO-OTFS systems. The proposed SBL-based Deep CNN (SBL-DCNN) is a modeldriven approach that combines the conventional expectation maximization (EM)-based SBL algorithm with a novel sparsity mask feature tailored exclusively for the delay Doppler domain (DD) to promote sparsity in channel predictions and accelerate the convergence of the training stage. This hybrid strategy improves MIMO-OTFS channel estimation by combining the strengths of SBL and domain-specific features, employing a cascade of diverse 2D convolution filters to effectively capture the complex underlying channel sparsity structures. The proposed network trained on data at moderate signal-to-noise power ratio (SNR) values demonstrates strong generalization and improved prediction performance across a wide range of SNR conditions in comparison to conventional sparse signal recovery-based channel estimation schemes.