Continuous, Contactless, and Multimodal Pain Assessment During Surgical Interventions

<i>Goal</i>: We introduce a continuous, multimodal pain classification technique that utilizes camera-based data conducted in clinical settings. <i>Methods</i>: We integrate facial Action Units (AUs) obtained from samples with sequential vital parameters extracted from video data, and systematically validate the practicality of measuring heart rate variability (HRV) from video-derived photoplethysmographic signals against traditional sensor-based electrocardiogram measurements. Video-based AUs and HRV metrics acquired from ultra-short-term processing are combined into an automated, contactless, multimodal algorithm for binary pain classification. Utilizing logistic regression alongside leave-one-out cross-validation, this approach is developed and validated using the BioVid Heat Pain Database and subsequently tested with our surgical Individual Patient Data. <i>Results</i>: We achieve an F1-score of 53% on the BioVid Heat Pain Database and 48% on our Individual Patient Data with ultra-short-term processing. <i>Conclusion</i>: Our approach provides a robust foundation for future multimodal pain classification utilizing vital signs and mimic parameters from 5.5 s camera recordings.