Multimodal hand/finger movement sensing and fuzzy encoding for data‐efficient universal sign language recognition

Abstract Wearable sign language recognition helps hearing/speech impaired people communicate with non‐signers. However current technologies still unsatisfy practical uses due to the limitations of sensing and decoding capabilities. Here, A continuous sign language recognition system is proposed with multimodal hand/finger movement sensing and fuzzy encoding, trained with small word‐level samples from one user, but applicable to sentence‐level language recognition for new untrained users, achieving data‐efficient universal recognition. A stretchable fabric strain sensor is developed by printing conductive poly(3,4‐ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS) ink on a pre‐stretched fabric wrapping rubber band, allowing the strain sensor with superior performances of wide sensing range, high sensitivity, good linearity, fast dynamic response, low hysteresis, and good long‐term reliability. A flexible e‐skin with a homemade micro‐flow sensor array is further developed to accurately capture three‐dimensional hand movements. Benefitting from fabric strain sensors for finger movement sensing, micro‐flow sensor array for 3D hand movement sensing, and human‐inspired fuzzy encoding for semantic comprehension, sign language is captured accurately without the interferences from individual action differences. Experiment results show that the semantic comprehension accuracy reaches 99.7% and 95%, respectively, in recognizing 100 isolated words and 50 sentences for a trained user, and achieves 80% in recognizing 50 sentences for new untrained users. image