CoEmo: Modeling Cognitive Processes in Facial Expression Recognition through Action Units and Gender Perspectives

Facial expression recognition lies at the intersection of computer science and cognitive psychology, yet the cognitive structure underlying facial action unit (AU) and emotion processing remains unclear. Are AUs and emotions processed in parallel or sequentially? Does gender influence this process? We constructed a 3D face dataset annotated with AU amplitudes and emotion labels. To model cognitive processing hypotheses, we implemented parallel and sequential architectures via multi-task learning and pipelined CNNs. Gender-specific models were compared using representational similarity analysis (RSA) with theoretical emotion spaces. The parallel model (F1 = 42.9%) outperformed the sequential one (F1 = 17.1%), supporting the parallel processing hypothesis. RSA revealed that females' emotion recognition aligned with social distance between emotions, while males' performance was selectively influenced by anger representations. These findings suggest sex-specific representational structures in emotion processing and support parallelism as a plausible cognitive mechanism in facial expression recognition.