Multimodal Pain Recognition Based on Contrastive Adversarial Autoencoder Pretraining

Background: Automated pain assessment aims to enable objective measurement of patients’ individual pain experiences for improving health care and conserving medical staff. This is particularly important for patients with a disability to communicate caused by mental impairments, unconsciousness, or infantile restrictions. When operating in the critical domain of health care, where wrong decisions harbor the risk of reducing patients’ quality of life—or even result in life-threatening conditions—multimodal pain assessment systems are the preferred choice to facilitate robust decision-making and to maximize resilience against partial sensor outages. Methods: Hence, we propose the MultiModal Supervised Contrastive Adversarial AutoEncoder (MM-SCAAE) pretraining framework for multi-sensor information fusion. Specifically, we implement an application-specific model to accomplish the task of pain recognition using biopotentials from the publicly available heat pain database BioVid. Results: Our model reaches new state-of-the-art performance for multimodal classification regarding all pain recognition tasks of ‘no pain’ versus ‘pain intensity’. For the most relevant task of ‘no pain’ versus ‘highest pain’, we achieve 84.22% accuracy (F1-score: 83.72%), which can be boosted in practice to an accuracy of ≈95% through grouped-prediction estimates. Conclusions: The generic MM-SCAAE framework offers promising perspectives for multimodal representation learning.