Iterative Detection for Orthogonal Time Frequency Space Modulation with ISTA

Orthogonal time frequency space (OTFS) modulation has emerged as an alternative to orthogonal frequency division multiplexing (OFDM) in high mobility scenarios due to its excellent performance in time-varying channels. With proper signal detection, the OTFS system can effectively utilize time and frequency diversity. However, most existing detection algorithms rely on message passing (MP), so their performance degrades significantly when the channel exhibits weak sparsity. The purpose of this paper is to propose a signal detection algorithm with excellent performance even in rich-scattering environments. We first model the OTFS signal detection problem as an optimization problem and an iterative shrinkage thresholding algorithm (ISTA)-based signal detection method is then proposed to solve the problem with better performance than the MP algorithm. Besides, we propose a modified fast iterative shrinkage-thresholding algorithm (FISTA)-based detection method by adopting an acceleration term to speed up iterative convergence. Numerical results show that the proposed algorithms have excellent convergence and BER performance.