Using Neural Networks to Better Understand Static and Dynamic Components of Facial Expression Recognition

Facial expressions are crucial social information for human communication. In the real world, facial expressions are dynamic; however, much of existing research in facial expressions relies on static stimuli. This static approach may limit the ecological validity of our understanding of the emotional information on faces. Our study aims to investigate the dynamic and static information contained in different emotional categories of facial expressions (e.g., happy, sad). Four convolutional neural networks are introduced as models to be trained with a large-scale dynamic facial expression dataset (short videoclips of 16 frames of 7 different emotional categories) in four different ways: ordered frames (of a single videoclip,16 at a time), shuffled frames, ordered global temporal change (15 optical flows or dynamic changes between adjacent frames, of a single videoclip in correct global sequence), and shuffled local temporal change. We compare model performance across these different training regimes in their ability to accurately identify the different emotional categories. Our results show that local dynamic information contributes to the recognition of sad and angry, and to a lesser extent fear, expressions, and global dynamic information contributes to the recognition of surprise and happy, and to a lesser extent disgust and fear, expressions. As expected, static structural information contributed to the recognition of sad, neutral, happy, surprise, and fear expressions. By highlighting the differential contributions of dynamic and static information, this study emphasizes the need for more ecologically valid approaches in the study of facial expression recognition.