SignPepper: Multimodal Social Robot for Sign Language Teaching

Sign language is an essential communication tool, however, it can be highly challenging for non-deaf students to learn. We propose a Pepper robot based sign language teaching assistant called SignPepper, with the ability to communicate in both spoken and sign language. Using Whisper speech to text and Llama 3.3, SignPepper can engage in two way spoken lessons, with the ability to physically demonstrate signs to students. Furthermore, using 3D convolutional neural networks trained on sign language recognition, SignPepper can watch, analyze and give personalized feedback on students attempts at performing newly learned signs in real-time; including hand-based error localization.