How Silent Are Silent Speech Interfaces? Speech Reconstruction From Whispered and Silent Ultrasound Tongue Images

<p>In the silent speech interfaces (SSI) area the aim is to restore or recognize speech whenever normal verbal communication is not possible or desirable. SSI systems use some non-acoustic biosignal of the body (e.g. tongue or lip movement) as input, and they are typically trained on data where real speech was produced, implicitly assuming that during silent (i.e. whispered or silently articulated) speech production the articulatory organs move similarly as they do during normal speaking. In this study we test this hypothesis in practice: we train our speech restoration DNNs on ultrasound tongue images recorded during audible speech, and synthesize speech from images recorded during whispering and two types of articulated-only speech. We found that synthesized speech for these silent "speaking" styles is significantly less intelligible than for audible speech, suggesting a difference in the articulatory movements, which should be considered when training silent speech restoration models.</p>