Randomized Modality Mixing with Patchwise RBF Networks for Robust Multimodal Pain Recognition

Pain recognition based on multimodal physiological signals remains a challenge, not only because of the limited training data, but also due to the varying responses of individuals. In this article, we present a randomized modality mixing technique (Modmix) for multimodal data augmentation and a patchwise radial basis function (RBF) network designed to improve robustness in limited and highly heterogeneous data. Modmix generates new samples by randomly swapping modalities between existing data points, creating new data in a very simple but effective way. The RBF patch network divides the input into randomly selected, overlapping patches that capture local similarities between modalities. Each patch network is trained end-to-end using stochastic gradient descent. Moreover, the model’s performance is further improved by using multiple independently trained networks and combining them into a single decision. Experiments with the two different pain datasets X-ITE and BioVid were performed under limited training data conditions, where only approximately 30% of the original datasets were used for training. With both datasets the RBF patch network achieved significant improvements for a subset of subjects, resulting in a similar or even slightly better mean accuracy compared to competing related models such as random forest and support vector machine.