Game Analysis Using Multimodal Emotion Recognition

Gaming industry is a very rapidly evolving domain where understanding player engagement is necessary for im-proving user experiences and content strategies. A multimodal framework for emotion recognition and engagement analysis in streamer videos is proposed, using both visual and audio data. The integration of facial and audio emotion recognition models are done to extract, classify and analyze emotions which is combined with a decision-level fusion for improved accuracy. The streamer's videos are preprocessed to extract facial frames and audio clips and separate CNN models are trained for each modality and decision fusion is implemented to combine the final predictions. Temporal emotional patterns, comparative emotional prevalence across streamers, and audio engagement levels are visualized using various graphs like emotion-over-time graphs, emotion distribution charts and engagement plots. The final results show that integrating both audio and visual en-hances the emotion detection accuracy which offers a comprehensive understanding of streamer behaviors and emotional impacts. Applications for this paradigm could include audience engagement measurement, content analysis, and emotion-driven insights for gaming industries and platforms.