Multimodal deep learning with particle swarm guided feature refinement for automated pain detection

Pain assessment remains a significant challenge in clinical practice because conventional self-reporting methods are subjective and unreliable for non-verbal, cognitively impaired, and critically ill patients. Existing automated pain detection approaches primarily rely on single-modality data or limited multimodal integration, resulting in inadequate feature representation, feature redundancy, and reduced classification performance. To address these limitations, this study proposes a multimodal deep learning framework that integrates physiological signals, facial images, magnetic resonance imaging (MRI), and clinical text for objective pain detection. The physiological branch employs a hybrid architecture comprising a Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) network, Bidirectional Gated Recurrent Unit (BiGRU), and Deep Neural Network (DNN) to learn hierarchical spatial-temporal representations. The facial branch combines EfficientNet and Residual Network (ResNet) to extract discriminative facial features, while a CNN-based architecture processes MRI images. Clinical narratives are encoded using Bidirectional Encoder Representations from Transformers (BERT) and BioBERT to capture contextual semantic information. An Adaptive Feature Refinement Layer (AFRL) enhances modality-specific feature learning, followed by Particle Swarm Optimisation (PSO) for feature selection and a novel PSO-Guided Feature Refinement Layer (PFRL) for adaptive feature weighting prior to multimodal fusion. We implement the framework in TensorFlow and PyTorch and evaluate it on the BioVid, UNBC-McMaster, and Trigeminal Neuralgia MRI datasets. Experimental results achieve 96.2% accuracy, outperforming Support Vector Machine (82.4%), K-Nearest Neighbors (79.6%), Bidirectional Long Short-Term Memory (88.9%), and AdaBoost (90.7%). The proposed framework provides a robust, scalable, and clinically applicable solution for automated multimodal pain detection and improved clinical decision support.