Deep learning-enhanced atomic norm minimization for DOA estimation of coherent and incoherent sources using coprime array

Abstract In the field of array signal processing, accurate direction of arrival (DOA) estimation is crucial for handling both coherent and incoherent mixed signal sources. This paper proposes a deep learning-enhanced atomic norm minimization (DLEANM) algorithm, designed to improve DOA estimation for coprime arrays. The algorithm first models the mixed signal scenarios realistically and employs interpolation to transform coprime arrays into virtual uniform linear arrays, which facilitates the formulation of the atomic norm minimization problem. Solving this problem reconstructs an augmented Hermitian Toeplitz covariance matrix. To further enhance the accuracy of DOA estimation, we developed a multi-scale convolutional neural network with an attention mechanism, which uses the covariance matrix as input to better capture signal variations across different frequencies and spatial distributions. By processing multi-scale inputs in parallel, the model improves adaptability and robustness in estimating mixed signal sources. Simulation results show that, under equivalent SNR and snapshot conditions, the DLEANM algorithm achieves lower estimation errors, more accurate multi-target DOA estimation, and relatively faster computation speed compared to conventional methods. Field experiments further confirm the effectiveness of the proposed algorithm. Therefore, after training, the algorithm is able to accurately identify the directions of unknown signals received by sensor arrays.