EchoLip: Pushing the Limit of Acoustic-Based Silent Speech Interface on Mobile Devices

Silent speech interface (SSI) enables users to interact with their devices without making audible sounds, thus preventing potential eavesdropping or disruptions to others. Recent advancements in acoustic sensing technology have made SSI on mobile devices highly promising, requiring no hardware modifications and operating in a non-contact manner. However, one major challenge faced by existing acoustic sensing-based SSI is its limited sensing range, typically less than 7cm. Users often need to speak in close proximity to the speakers/microphones, severely constraining its applicability on mobile devices. In this paper, we introduce EchoLip, which can significantly increase the sensing range and enhance long-term usability in real-world settings. EchoLip utilizes the smartphone’s two built-in speakers and microphones to transmit and receive mutually orthogonal wave signals to capture multi-view information. Then, a specially designed signal processing pipeline and neural network are used to extract fine-grained features that adapt to different angles and distances. We also design a lip movement monitoring algorithm to handle various interference. We evaluate EchoLip on 20 individuals using a set of 500 sentences. EchoLip achieves an average Word Error Rate of 13.9% and 19.7% at 15cm and 40cm. Evaluations in various scenarios further validate the robustness of EchoLip.