Audioradio Target Speech Detection and Extraction with mmWave Sensing

Detecting the target speaker and enhancing speech with high fidelity has been a long-standing problem, especially in challenging acoustic conditions. To address these problems with minimal user cooperation, we have developed multimodal audioradio speech detection (RadioVAD) and enhancement (RadioSES) systems using mmWave modality. This demo presents how these two systems can be run together in real time to extract target speech and filter any type of ambient noise. Our demo uses an mmWave radar and a microphone attached to a laptop to detect and localize target speakers in the field of view, detect the presence of voice to trigger the microphone, and enhance the noisy speech with a deep learning model running in real-time. Our experiments confirm that an audioradio system can detect and isolate high-fidelity target speech, even with interfering speech and noise; while being privacy preserving and environmentally robust.