Multimodal Model for Automated Pain Assessment: Leveraging Video and fNIRS

Pain assessment is a challenging task for clinicians due to its subjective nature, particularly in individuals with communication difficulties, cognitive impairments, or severe disabilities. Traditional methods such as the Visual Analogue Scale (VAS), Numerical Rating Scale (NRS), and Verbal Rating Scale (VRS) rely heavily on patient feedback, which can be inconsistent and subjective. To address these limitations, developing objective and reliable pain assessment tools that incorporate advanced technologies, such as multimodal data integration from video and fNIRS, is important for improving clinical outcomes. However, challenges such as noise susceptibility in fNIRS signals must be carefully addressed to realize their full potential. Recent studies have explored automatic pain assessment using machine learning and deep learning techniques, which require high-quality data that can accurately represent pain categories. In response to the introduction of a new dataset in the AI4Pain Challenge, we proposed a multimodal neural network model utilizing attention-based fusion to improve overall accuracy (MMAPA). Our model leverages video and fNIRS modalities as well as manually extracted statistical features. We also implemented fNIRS signal preprocessing and artifact noise filtering, which significantly improved performance on both the fNIRS and statistical feature branches. On the hidden test set, our model achieved an accuracy of 51.33%, outperforming the official baseline of 43.33%. To evaluate generalizability, we further tested our method on the BioVid Heat Pain Database, where our fusion model achieved the highest accuracy in the 10-fold cross-validation setting, outperforming PainAttNet and unimodal variants. These results highlight the effectiveness of our multimodal attention-based approach in improving pain classification performance across datasets.