A Multimodal Emotion Analysis System for Psychological Counseling Using Image, Audio, and Large Language Model Inference

Traditional psychological counseling methods often suffer from a lack of emotion quantification, discontinuous records, and overreliance on subjective judgment, hindering effective tracking of patients’ emotional dynamics. Current multimodal emotion recognition systems also face challenges such as misaligned emotional cues and conflicts between modalities due to temporal and expressive inconsistencies, with no reliable mechanism for assessing cross-modal consistency. To address these issues, we propose Image-Audio-LLM-based Multimodal Emotion Analysis System (IALLMEAS), a novel psychological counseling support system that integrates imageaudio emotion recognition with large language model (LLM) reasoning. The system extracts facial and vocal emotions through dedicated models and generates fine-grained temporal emotion trend tables. Prompt engineering guides the LLM to evaluate modality reliability, detect noise or conflicts, and perform causal reasoning.IALLMEAS produces interpretable, causally-informed therapeutic reports, bridging information gaps in traditional counseling and highlighting key emotional transitions. This helps clinicians devise more targeted and sustained interventions. Experimental results show that IALLMEAS achieves high accuracy in emotion recognition and strong multimodal consistency, providing robust support for efficient, personalized, and traceable psychological care.