Indian Sign Language Detection in Virtual Reality using Leap Motion Controller

Sign Language is a powerful tool that helps people with hearing and speech impairment to effectively communicate with normal people in real environment. Such people should also be able to communicate in virtual environments. The integration of Sign Language and Virtual Reality holds promise for breaking down communication barriers, empowering deaf individuals, and advancing accessibility and inclusion in virtual environments. It can enable them to fully participate in immersive environments. This paper focuses on the use of Leap Motion Controller and Virtual Reality devices for Indian Sign Language recognition, with the intention of increasing accessibility. Leap Motion Controller is used to capture Indian sign language gestures, that can be interpreted to text and thus providing people with a natural and expressive way to interact in Virtual Reality environments. The predicted alphabets and gestures are displayed in a Virtual Reality environment for visualization and better communication. EfficientNetB3, ResNet50 and Inception v3 are used for training the sign language data and prediction of correct gestures. Out of these models, EfficientNetB3 and Inception v3 achieved accuracies of 0.9984 and ResNet50 had an accuracy of 0.9987.