Triboelectric E-Skin Patch Decoding Silent Speech for Advanced Human–Machine Interaction and Augmented Virtual Communication

Abstract Silent speech decoding (lip reading) represents an innovative hands-free communication method for individuals with laryngeal and lingual impairments, addressing critical daily communication challenges. Herein, a novel silent speech decoding system is proposed, which employs a self-powered, low-cost, and conformable triboelectric-based E‑skin patch and a well-trained multi-layer convolutional neural network (CNN) model. We systematically characterize the structural principles and electrical performance of the E-skin patch. A comparative analysis of labial kinematics during vocalized and silent speech is also conducted, which demonstrates strong consistency in time–frequency domain characteristics. The utilized CNN model is trained across 15 lexical categories (200 samples per category), achieving a high accuracy of 98.75% and demonstrating excellent recognition performance. Furthermore, by integrating human–machine interaction (HMI) and virtual reality (VR) technologies, we develop three representative applications, including a real-time lip language visualization system, an orofacial-controlled “Feeding Frenzy” interactive game, and a VR communication platform with avatar dialogue. This technology enables barrier-free communication for voiceless populations, demonstrating significant potential in HMI, augmentative communication systems, virtual reality integration, and biomedical rehabilitation.