A Deep Learning Approach to Assess Social Skills in Virtual Reality Based Interventions for Children with Autism Spectrum Disorder

Autism Spectrum Disorder (ASD) children struggle with social communication and Theory of Mind (ToM), making it difficult to recognize other people's thoughts and feelings. They may face challenges comprehending nonverbal cues, making eye contact, and initiating conversations. These difficulties tremendously affect the ability to connect and sustain relationships effectively with others. Conventional training methods may not be sufficient to improve their ToM and social skills. These approaches do not offer immersive and context-rich scenarios to practice in a dynamic social environment. Conventional training methods can be enhanced by integrating Virtual Reality (VR) which provides a more engaging, safe, and interactive setting. Two VR scenarios were created for autistic children to improve specific social behaviors like making eye contact and waving hands. These scenarios were created with Unity 3D to foster interaction during a comprehensive Social Skill Training (SST) approach. Data were acquired when the children responded in both instances by raising their hands and not. A U-Net architecture segments the hand region, enabling accurate recognition of hand positions and movements. Following that, the customized Convolutional Neural Network (CNN) is employed to assess the responses of social skills in VR-based interventions. The suggested CNN has accomplished a training accuracy of 94.9% and a validation success rate of 96.3% for identifying the presence or absence of a child's hand as raised or not raised. The efficiency of the proposed model is estimated by using different pre-trained models. The results reveal that the highest accuracy was attained on the augmented dataset compared to the original dataset.