Ensemble Deep Learning Framework for Hybrid Facial Datasets Using Landmark Detection: State-of-the-Art Tools

Autonomous face emotion recognition (FER) with landmarks has become an important field of research for human–computer interaction. A significant achievement has been achieved through deep learning algorithms in recent times. Recognizing faces can be done using an end-to-end approach with deep learning techniques, which learns a mapping from raw pixels to the target label. In the field of emotional classification, the research community has extensively utilized 98 and 68 facial landmarks. In particular, pre-trained convolutional neural networks such as the residual network 50-layer network with the random sampler, Visual Geometry Group 16-layer network, and MobileNet including their ensemble versions of deep learning models are popular among researchers due to their ability to handle complex data. Researchers have mostly evaluated the model on a single dataset. A single dataset poses a challenge in developing a generalized model capable of capturing the full versatility of emotions. The key challenge in the dataset is that a single emotion is represented in multiple facial expressions with low-resolution images. This research study uses a combined dataset (CK+, KDEF, and FER-2013), which is more challenging than a single dataset. This research study offers a comprehensive analysis involving 68 and 98 landmarks with different FER deep models, examining how landmarking and different network architectures contribute to emotion recognition accuracy. This research study also considers the overfitting and class imbalance of the proposed ensemble model, which improves its performance by batch-wise feature extraction. Results show 78% accuracy with 98 landmarks and 75% with 68 landmarks. Overall, the model significantly reduces the gap between training and testing accuracy for both single and combined datasets. Received: 28 September 2024 | Revised: 13 January 2025 | Accepted: 13 February 2025 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement The CK+ data that support the findings of this study are openly available in Kaggle at https://www.kaggle.com/datasets/shuvoalok/ck-dataset. The KDEF data that support the findings of this study are openly available in Kaggle at https://www.kaggle.com/datasets/tom99763/testtt. The FER2013 data that support the findings of this study is openly available in Kaggle at https://www.kaggle.com/datasets/msambare/fer2013. Author Contribution Statement Nandhini Krishnasamy: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization, Supervision, Funding acquisition. Nilima Zade: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization, Supervision, Project administration. Dhruvi Khambholia: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Rabinder Henry: Validation, Formal analysis, Data curation, Writing – review & editing, Visualization, Supervision, Project administration, Funding acquisition. Aditya Gupte: Validation, Formal analysis, Data curation, Writing – review & editing, Visualization, Supervision, Project administration.