Single-Channel Speech Enhancement with Normalizing Flows

Speech Enhancement (SE) denotes a family of techniques with the goal to improve quality and intelligibility of a speech signal. A main task of SE is to remove background noise from corrupted recordings and it has been studied for several decades. SE experienced great recent progress along with the success of Deep Neural Networks (DNNs) with methods based on Discriminative Models (DMs) and Generative Models (GMs). DMs for SE often use multiplicative separation masks applied to a time-feature representation of the noisy input signal. However, DMs regularly struggle to show satisfactory results in difficult scenarios, such as low Signal-to-Noise Ratios (SNRs), as their performance is often limited by the number of speech components that can be extracted from the noisy input. GMs for SE provide an alternative, as they learn the fundamental structure of clean speech utterances as captured by their probability distribution, which can then be used to overcome the aforementioned issues. Normalizing Flows (NFs) are one family of GMs and are based on invertible and differentiable transformations. They allow for direct and efficient modelling of Probability Density Functions (PDFs) using DNNs. Although NFs showed a large potential in related fields, it was not well understood yet how they perform in SE. The contributions in this thesis examine the application of NFs for single-channel SE and discuss its benefits and shortcomings. First, the so-called Speech Enhancement Flow (SE-Flow) model is proposed as the main network architecture employed in the contributions. The model takes clean and noisy speech utterance pairs in the waveform domain to estimate a conditional PDF of clean speech samples. The corresponding experimental evaluation on a common SE data set confirms the noise suppression capability of this approach, especially when compared to Generative Adversarial Network (GAN)-based baselines. Moreover, the usage of non-linear companding is proposed as a pre-processing technique to the input signals. This setup demonstrates a stronger SE performance compared to using the pure waveform signal in the experimental evaluation. Second, several modifications of the SE-Flow architecture are proposed to improve the SE performance. The initial SE-Flow model is based on a so-called coupling flow, where only one part of the input signal is transformed in each network building block. This structure is beneficial to maintain invertibility and computational efficiency, but also requires a large number of blocks to achieve the targeted outcome. Therefore, a modification of the structure of these building blocks is proposed to transform the entire input signal in one step. Moreover, perceptually motivated pre-processing representations are investigated to guide the model towards generating better output signals. According to perceptual evaluation, the specifically developed All-Pole Gammatone filter-bank representation performs best among comparing GAN and Denoising Diffusion Probabilistic Model approaches especially in low SNR conditions. Third, although the proposed changes in the SE-Flow model improved the overall performance, a gap to more recently developed GMs for SE remained. Moreover, simply increasing the network complexity in terms of more layers and parameters did not lead to further performance gains. In order to achieve additional improvements, more adaptions to the model are proposed. The main adjustment consists in changing the original pre-processing of the conditional side information from a single convolutional layer for each individual network block to a specific conditioning network. Therefore, the encoder of a U-Net DNN is used as a more effective and meaningful feature extractor, as it takes previous processing steps into account. The experimental evaluation demonstrates that the conditioning network leads to higher performances in a perceptual evaluation and computational metric results. GANs are known of being capable in producing very realistic samples for SE. However, they require lots of engineering and do not outline a properly defined PDF. As a solution, a new model called Speech Enhancement Flow GAN (SEFGAN) is proposed, as a combined framework of a NF-based SE model and a GAN. Therefore, the trained and inverted flow model serves as a generator in the GAN framework and learns to generate samples with better perceptual results. Comparing this technique to other well-established methods in generative SE, SEFGAN achieves competitive perceptual and computational metric results, while requiring less computational load. Moreover, a hybrid approach overcomes the loss of likelihood-estimation capability, which is common among GANs, while maintaining a leading performance. In summary, the contributions in this thesis represent the first NF-based framework for single-channel SE that directly incorporates the noisy speech utterance into the model. Overall, the combination of efficiency and high-quality speech estimates delivered by the proposed method is unmatched by comparing models and provides a betterunderstanding of the field of generative SE.