MiEmo: A multi-modal platform on emotion recognition for children with autism spectrum condition

This study presents MiEmo, a multi-modal digital platform designed to improve emotion recognition in children with Autism Spectrum Condition (ASC). The platform integrates serious games with music and color as feedback mechanisms to strengthen emotion understanding in addition to traditional visual interventions such as pictures and videos. The study aims to assess the usability and potential effectiveness of MiEmo in supporting therapy for children with medium- and high-functioning ASC. A pilot usability study was conducted in two rehabilitation centers involving 19 children, 8 with medium and 11 with high functioning ASC. Participants engaged with six training activities, or exergames, on the MiEmo platform that implemented multi-modal feedback (music pieces and colored animations associated to the emotion). The System Usability Scale (SUS) and qualitative feedback from therapists were used to evaluate the platform's usability. The average SUS scores were 86.88 for children with medium-functioning ASC and 96.75 for those with high-functioning ASC, indicating positive usability. Therapists noted that while the platform was well-received, further updates are needed for better adaptation to medium-functioning children. Multi-modal feedback, particularly music and color, was found to enhance emotion recognition, with children responding well to the integration of these sensory cues. The study demonstrates that MiEmo has significant potential as a tool for socio-emotional training, particularly for high-functioning children. However, limitations such as the small sample size, short intervention duration, and lack of a control group suggest that future studies with larger participant groups are necessary to validate these findings and assess long-term effects. • Digital applications can be useful in socio-emotional training for ASC children. • Music and colors multimodal feedback to enhance the emotion perception is largely unexplored in ASC training. • Music and color multimodal feedback is the core of the MiEmo digital platform for ASC training on emotion recognition. • Usability Testing with 19 children with ASC from two centers for ASC (R1 and R2) resulted in high acceptance. • Children in the study started to make confidences to therapists on what made them feel like the emotions portrayed on screen.