A survey of deep learning for complex speech spectrograms

Recent advancements in deep learning have significantly impacted the field of speech signal processing, particularly in the analysis and manipulation of complex spectrograms. This survey provides a comprehensive overview of the state-of-the-art techniques leveraging deep neural networks for processing complex spectrograms, which encapsulate both magnitude and phase information. We begin by introducing complex spectrograms and their associated features for various speech processing tasks. Next, we examine the key components and architectures of complex-valued neural networks, which are specifically designed to handle complex-valued data and have been applied to complex spectrogram processing. As recent studies have primarily focused on applying real-valued neural networks to complex spectrograms, we revisit these approaches and their architectural designs. We then discuss various training strategies and loss functions tailored for training neural networks to process and model complex spectrograms. The survey further examines key applications, including phase retrieval, speech enhancement, and speaker separation, where deep learning has achieved significant progress by leveraging complex spectrograms or their derived feature representations. Additionally, we examine the intersection of complex spectrograms with generative models. This survey aims to serve as a valuable resource for researchers and practitioners in the field of speech signal processing, deep learning and related fields. • Presents a comprehensive survey on deep learning for complex speech spectrograms. • Summarizes the key components (activation functions, batch normalization) and architectures (fully-connected layer, CNN, RNN, transformer) of complex-valued neural networks, which are designed for or have been used on complex spectrogram processing. • Summarizes the architectural designs of real-valued neural networks for complex spectrogram processing. • Summarizes low-latency techniques, training strategies (mapping vs masking) and loss functions (phase, complex, time and multi-domain loss functions) for learning complex spectrograms. • Highlights recent progress in phase retrieval, speech enhancement and separation using complex spectrograms. • Investigate the intersection of complex spectrograms with generative learning models, covering variational auto encoders, generative adversarial networks, and diffusion models.