Systematic Review of Image‐Based Emotion Recognition Datasets: A Comprehensive Analysis

ABSTRACT The present systematic review gives a broad overview of the image‐based datasets on which emotion has been classified. These datasets are categorized into three groups: (1) those containing only facial images, (2) those incorporating facial images with additional contextual body features, and (3) those based on visual components with physiological or audio signals. This paper discusses datasets in each category, focusing on their structure, emotion categories, emotion sources, anonymization level, and suitability for machine learning applications. A key aspect of this study is mapping emotions in the datasets analyzed to Plutchik's Wheel of Emotions, allowing standardization based on a consistent psychological model. This alignment demonstrates the need for more thorough and balanced datasets encompassing a wider range of affective states and exposes discrepancies in the representation of emotions. Comprehensive statistics on each dataset are also provided in our study, allowing for comparisons on sample size, distribution of emotions, anonymization, and sources of emotion. To conclude, this study discusses future directions, for which multimodal datasets that incorporate images, audio, and biosignals are of high importance as are databases containing a larger variety of emotions mapped more evenly across Plutchik's model. This article is categorized under: Technologies > Classification Technologies > Artificial Intelligence Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction