Automated pain detection using facial expression in adult patients with a customized spatial temporal attention long short-term memory (STA-LSTM) network

Self-reported pain scores are often used for pain assessments and require effective communication. On the other hand, observer-based assessments are resource-intensive and require training. We developed an automated system to assess pain intensity in adult patients based on changes in facial expression. We recruited adult patients undergoing surgery or interventional pain procedures in two public healthcare institutions in Singapore. The patients' facial expressions were videotaped from a frontal view with varying body poses using a customized mobile application. The collected videos were trimmed into multiple 1 s clips and categorized into three levels of pain: no pain, mild pain, or significant pain. A total of 468 facial key points were extracted from each video frame. A customized spatial temporal attention long short-term memory (STA-LSTM) deep learning network was trained and validated using the extracted keypoints to detect pain levels by analyzing facial expressions in both the spatial and temporal domains. Model performance was evaluated using accuracy, sensitivity, recall, and F1-score. Two hundred patients were recruited, with 2008 videos collected for further clipping into 10,274 1 s clips. Videos from 160 patients (7599 clips) were used for STA-LSTM training, while the remaining 40 patients' videos (2675 clips) were set aside for validation. By differentiating the polychromous levels of pain (no pain versus mild pain versus significant pain requiring clinical intervention), we reported the optimal performance of STA-LSTM model, with accuracy, sensitivity, recall, and F1-score all at 0.8660. Our proposed solution has the potential to facilitate objective pain assessment in clinical settings through the developed STA-LSTM model, enabling healthcare professionals and caregivers to perform pain assessments effectively in both inpatient and outpatient settings.