Are You Paying Attention? Multimodal Linear Attention Transformers for Affect Prediction in Video Conversations

The post-COVID-19 era has seen continual adoption and reliance on video-based communication, underscoring the need for unobtrusive affect recognition in digital interactions. This paper proposes an efficient multimodal approach to emotion recognition in video conversational scenarios, leveraging linear attention-based Transformer networks to process both visual and audio cues. We explore various linear attention mechanisms, comparing them with classical self-attention. Using the K-EmoCon dataset, we demonstrate that the proposed approach yields competitive performance in predicting the affective states of conversing persons while significantly improving memory efficiency. Our ablation studies reveal that carefully tuned simple fusion methods can match or exceed more complex approaches. This research contributes to developing more accessible and efficient multimodal emotion recognition systems for video-based conversations, with applications for enhancing remote communication and monitoring digital well-being in the post-pandemic era.