Human and AI Perspectives: Comparing Emotion Recognition from Facial Expressions and Hand Gestures

Abstract Emotional recognition plays a pivotal role in understanding human interactions, shaping decision-making processes, and enhancing communication across diverse domains. With technological advancements, there is an increasing demand for systems that can accurately interpret emotional cues, driving innovations in healthcare, marketing, and human-computer interaction. This study addresses the gap between human intuition and algorithmic analysis by evaluating the performance of human annotators and machine-learning algorithms in emotion recognition tasks. Utilizing the CMU, CAM3D, and BEAST datasets, we investigated the capabilities of human annotators to identify emotions from facial expressions and hand gestures. Concurrently, we applied the YOLOv8 algorithm to the same datasets to assess its efficacy under three distinct scenarios. Performance metrics including accuracy, precision, recall, F1-score, and mAP were employed to rigorously evaluate and compare the results. Our findings indicate that YOLOv8 achieves high accuracy on less complex datasets, such as CAM3D, but encounters difficulties with the nuanced and compound emotions present in the CMU dataset. This underscores the inherent challenges in emotion recognition tasks and highlights the potential of integrating multimodal inputs and human expertise to enhance machine-learning systems. Recommendations include diversifying datasets and fostering interdisciplinary approaches to address the algorithmic limitations. This research contributes substantially to machine learning by addressing a critical application problem: the development of accurate, context-aware emotion recognition systems. By offering empirical insights and methodological advancements, this study establishes a foundation for improving the classification, prediction, and real-time application of learning methods in fields such as healthcare, robotics, and virtual reality. The results not only advance computational approaches to learning but also demonstrate their impact on solving real-world challenges, aligning with the journal’s commitment to robust empirical studies and practical applications.