Electrospun Surface-Modified Epidermal Strain Sensors Enable Silent Speech and Hand Gesture Recognition for Virtual Reality Interaction

Voice disorders severely limit verbal communication, creating a need for intuitive assistive technologies. To meet this need, we present epidermal strain sensors that capture strain signals during silent speech and hand gesture. A thin electrospun nanofiber layer integrated onto commercial polyurethane films guides uniform, controlled microcrack formation in screen-printed carbon conductive paths, achieving a gauge factor up to 243 over 0–40% strain. Signals from the seven-channel strain sensor array are recognized by a hybrid neural network that combines convolutional and Transformer architectures, reaching over 98% accuracy. The recognized outputs are rendered in virtual reality (VR), enabling intuitive, real-time communication. Moreover, the approach simplifies fabrication by enabling crack-based strain sensing with only a thin electrospun surface layer on commercial polyurethane films, eliminating the need for thick freestanding electrospun substrates. This cost-effective approach addresses limitations of conventional electrospun substrates by minimizing the thickness of the electrospun layer, thereby shortening the electrospinning time. Overall, the work demonstrates a method for translating natural non-verbal expressions into speech and text in VR, with promising applications in healthcare and assistive communication.