Deep Canonical Correlation Analysis Convolutional Neural Networks for neonatal pain facial expression recognition

To deeply explore the intrinsic relationship between static texture features and dynamic muscle movements in neonatal pain videos from a multi-dimensional spatiotemporal perspective, and to establish the spatiotemporal collaborative relationship in local pain-sensitive regions of neonates, this paper proposes two artificial intelligence models for neonatal pain facial expression recognition: a Two-stream Deep Canonical Correlation Analysis Convolutional Neural Network (TDC-CNN) and a Two-stream Deep Canonical Correlation Analysis Cross Convolutional Neural Network (TDCC-CNN). Specifically, TDC-CNN embeds canonical correlation analysis (CCA) computation modules between convolutional blocks at each level of the two-stream architecture based on the Visual Geometry Group 16-layer network (VGG16). It maximizes the correlation between static texture features from the spatial stream and dynamic optical flow features from the temporal stream through gradient optimization, enhancing cross-stream correlated representation learning. Furthermore, TDCC-CNN presents a Deep CCA Cross module which utilizes CCA-aligned projected features to generate channel attention weights, driving bidirectional collaborative enhancement of spatiotemporal dual-path features. Simultaneously, through the stepwise addition of these modules across different network levels, multi-level spatiotemporal feature alignment is achieved, facilitating the learning of complementary spatial and temporal representations of neonatal pain expressions. Both models undergo extensive evaluation on the Video-based Facial Expressions of Neonatal Pain (VFENP) dataset and validation on an adult facial expression video database. Experimental results show that the two proposed models can efficiently fuse dual-path features, thereby improving all aspects of model performance, achieving results superior or comparable to those of state-of-the-art methods across all evaluation metrics. The proposed two models can be used for objective and continuous neonatal pain assessment based on artificial intelligence technology, assisting clinical medical decision-making.