The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds

Lip language is a straightforward method of non-verbal communication for people with laryngeal and lingual injuries, dysphonia, or vocal cord lesions. It is convenient for these speech-impaired individuals, especially when their hands are occupied. However, it depends mainly on line-of-sight-required sensors (e.g., cameras for vision-based and speaker/microphone for acoustic-based solutions) or specialized peripherals to collect and understand the lip language. Therefore, the usage of lip language is limited. Recently, inertial measurement units (IMUs) have been increasingly embedded among commercial off-the-shelf (COTS) earbuds (e.g., Apple AirPods). The built-in IMUs have the potential to sense silent articulatory gestures. However, these sensors present low sampling rates (of merely 25 Hz) and signal-to-noise ratios, blocking error-free lip reading. As a countermeasure, we comprehensively analyze the articulations of speech-impaired persons. Accordingly, we map inertial signals to lip language with effective noise cancellation. We propose an inertia-based lip language decoding system, named SwanTrumpet. It supports lip reading, unvoiced interaction, and user identification for speech-impaired users. In addition, we address the issue of user diversity in lip language translation, which is significant among the speech-impaired. We implement the prototype connected with two COTS earbuds. Extensive evaluations on 11 speech-impaired participants demonstrate its capability to improve the interaction experience for people lacking a voice.