Bluemarble: Bridging Latent Uncertainty in Articulatory-to-Speech Synthesis with a Learned Codebook

In this letter, we present Bluemarble, a novel neural articulation-to-speech (ATS) system designed to generate high-quality speech from articulatory recordings. Traditional ATS methods often demand a substantial dataset of corresponding articulatory and speech signal pairs for effective model training. This requirement arises because these approaches treat the estimation of acoustic features as a regression task, which can pose difficulties for accurately mapping latent articulatory features to the desired speech-related features. To tackle this challenge, we propose a method that involves employing a finite learned codebook to constrain the uncertainty latent space. Operating within a framework consisting of a speech mel-spectrogram encoder, quantizer, and decoder, our model establishes a mapping rule between the latent embeddings derived from electromagnetic articulography (EMA) signals and quantized discrete speech tokens. During the inference stage, EMA embeddings are first transformed into speech-related discrete tokens, which are then input into a neural vocoder to synthesize the speech waveform. Experimental results show that our approach outperforms existing state-of-the-art methods in both qualitative and quantitative assessments. Audio samples are available online . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>