SpDiff: A Speech Sensing System with Diffusion Model Based on mm Wave Radar

Voice control has become an indispensable interaction method in smart devices. Compared to traditional microphones, mm Wave radar offers a promising solution for speech sensing in noisy environments. However, most current research relies on single-view information, such as vocal cord vibrations or lip movements, to classify speech, which overlooks important details like timbre, speech rate, and intonation, limiting the application of speech sensing. To address these issues, we develop a high-quality speech sensing method based on mm Wave radar, named SpDiff. This method accurately localizes the vocalizing target and, based on the human vocal mechanism, extracts multi-view speech features according to the movement characteristics of the vocal cords, lips, and face. Additionally, to generate high-quality speech signals, we design a conditional latent diffusion model (CLDM), which uses multi-view radar information as conditional guidance, accurately capturing the complex mapping relationships between radar and speech signal distributions. To evaluate the SpDiff method, we build a mmWave system using IWR1443Boost and recruit 14 volunteers to construct a dataset. Experimental results show that SpDiff achieves high standards in speech sensing, with the generated speech directly input into existing recognition models, achieving an average character and word error rate (CER/WER) of only 2.33% and 3.05%.