Emotion-aware cross-modal music generation based on multimodal emotion recognition

This study presents a Multimodal Emotion-guided Music Generation Model (MEMGM) that directly translates multimodal affective states into instrumental music. Unlike prior approaches relying on textual intermediaries or discrete categories, MEMGM integrates facial expressions, speech prosody, and ECG-derived heart signals into a continuous valence–arousal–dominance (VAD) space for fine-grained emotion representation. A new MIDI-based dataset with continuous VAD annotations is constructed, enabling controllable manipulation of musical elements and nuanced mapping of emotional dynamics. The framework combines a cross-modal Transformer, multimodal BiLSTM, and gating block for robust affective recognition, followed by a plug-and-play generation module enhanced with supervised contrastive learning to ensure intra- and inter-modal consistency. Experimental results demonstrate superior performance in both emotion recognition and music generation compared to state-of-the-art baselines. Subjective evaluations confirm that the generated music aligns closely with users’ perceived emotions. The main contributions are: (1) an end-to-end pathway from multimodal signals to music without textual intermediaries, (2) a VAD-annotated MIDI dataset for expressive generation, and (3) a contrastive learning strategy that enhances affective alignment across modalities.