Mobile-Based Serious Game for Learning Facial Expression Recognition in Children with Autism Spectrum Disorder Using MobileNet Model

The human ability to understand someone's emotions through facial expressions plays a crucial role in everyday social interactions. However, in children with autism spectrum disorder (ASD), this ability is often impaired, affecting their social interactions. Studies have shown that children with ASD are well-acquainted with and often engage with technology, including smartphones and video games. This research proposes a solution in the form of a mobile-based serious game application as an educational tool to assist children with autism in learning human emotions through facial expressions. Serious games are considered appropriate educational tools because it leverages four pillars of learning: attention, active learning, feedback, and consolidation. The gamification technique used involves a snake and ladder game that prompts users (children with autism) to mimic the facial expressions displayed in the game. The game also measures the accuracy of the user's facial expressions using MobileNet model, then uses it as scores within the gameplay to enhance user engagement. This model reached the highest validation accuracy of 0.89 with training accuracy of 0.88. Final evaluation of the model is conducted using test set and resulted in an accuracy of 0.83.