MDD-FPA: A Multimodal Depression Detection Method Based on the Fusion of Pleasure and Arousal

Multimodal depression detection shows promising research prospects by combining audio, textual, and visual data. However, existing multimodal fusion methods usually ignore the variability in the contribution of different modalities in depression detection, leading to an imbalance of inter-modal information during feature fusion. To mitigate this issue, we propose a Multimodal Depression Detection method based on the Fusion of Pleasure and Arousal (MDD-FPA). First, we introduce the concepts of pleasure and arousal to quantify the tendency and degree of depressive affect. Based on this, we regard weak modality features as a reference value and construct pleasure and arousal features with other features. Finally, we realize a balanced fusion of intermodal information by fusion strategies in horizontal and vertical directions. In addition, we design a feature extractor named SER-BiGRU, which combines Squeeze-and-Excitation Networks(SEnet) and Bidirectional Gated Recurrent Units(BiGRU) to enhance the mining ability of depression-related information. The findings of the experiment demonstrate that the proposed MDD-FPA significantly outperforms baseline approaches on the DAIC-WOZ depression dataset, which validates the effectiveness of the method in deep depression information capture and multimodal feature fusion.