PM-ViT a Framework for the Recognition of Emotions and Proclivity toward Mental Illness Using Facial Expressions

Automated emotion recognition is being used as a powerful technology in various fields in the present times. Facial Emotion Recognition (FER) aims to identify the emotional states of individuals based on their facial expressions. While in recent years, Convolutional Neural Networks (CNNs) have shown noteworthy performance in image classification tasks, however, the latest adoption of transformers for computer vision tasks has become really influential. This study proposes a novel ViT-based framework PM-ViT to explore the performance of Visual Transformer (ViT) based models on emotion recognition tasks and compare its performance with a CNN-based approach and other existing ViT-based models that recognize emotions from images. The proposed model PM-ViT takes facial images as input. It recognizes the expression and does a binary classification into two classes negative emotions and positive emotions. In addition to emotion recognition, in case the emotions found are negative PM-ViT does a further classification in three classes mild, moderate, and severe basis the perceived strength of negative emotion and hence the proclivity that the person may be having a mental illness. The experimental findings demonstrate that the model using CNN achieves an F1-score of 81.0% on AffectNet and 97.8% on the CK+ augmented dataset whereas the proposed PM-ViT achieves an F1-score of 84.0% on AffectNet and 99.7% on CK+ augmented dataset. The performance of PM-ViT surpasses the performance of the existing ViT-based techniques that determine emotions from images.