Multi-Domain Decoupled Bayesian Channel Estimation for Massive MIMO-OTFS Systems

Orthogonal time frequency space (OTFS) modulation has shown exceptional performance in high-mobility scenarios. However, downlink channel estimation in massive multiple-input multiple-output (MIMO) OTFS systems remains challenging due to limited Doppler and angular resolutions, underutilization of domain-specific channel characteristics, and substantial computational complexity. In this paper, we propose a multi-domain decoupled channel estimation framework that separates the coupled channel into its individual domain components. The framework enhances the utilization of channel characteristics and provides more flexibility in algorithm design. Based on the proposed framework, we develop a multi-domain decoupled Bayesian learning (MDDC-BL) channel estimation algorithm, achieving super-resolution performance in both Doppler and angular domains. For angular-domain estimation, we devise a dictionary learning based pattern-coupled Bayesian learning (DL-PC-BL) channel estimation algorithm to effectively capture the channel characteristics and handle the off-grid components. Additionally, the MDDC-BL algorithm reduces the dimensionality of the estimation model, thereby significantly lowering computational complexity. Simulation results demonstrate the superior performance of the proposed framework and channel estimation algorithms over existing state-of-the-art schemes.