A Reliability Sensitive Theory of Multimodal Emotion Recognition

Current research on multimodal emotion recognition seeks to analyse the relationships among facial, vocal, and bodily expressions in order to recognise different emotions. Although additional data streams are often introduced, they cannot be treated as independent evidence. This conceptual study aims to understand the reliability of multimodal inferences over unimodal classifications. This study covers the theory and practice of affective computing, the study of different modalities, fusion taxonomies and benchmarking. Depending on the statistical dependencies that they learn, fusion can occur in early, intermediate, late or adaptive form. The synthesis suggests that the advantage of multimodal systems is determined by four factors: informational complementarity, partially independent errors, temporally valid alignment, and dynamic reliability estimation. For these factors, fusion of different types of data is a good alternative. Otherwise, it is likely to increase the recording of social displays, bias of annotations, or dependency on a dominant modality. For this reason, a reliability-sensitive framework has been proposed, in which recognition is driven by uncertainty management rather than feature accumulation, while preserving the modality-specific nature of different types of evidence. The resulting account explains both the recurrent gains reported in benchmark studies and their contraction under naturalistic expression, distribution shift, cultural mismatch, strategic display, and systematic missingness. Multimodal systems become defensible not by sensing more behaviour alone, but by estimating when each observation should change, confirm, or weaken an affective inference.