Multimodal emotion recognition and feedback adaptation for personalized music therapy

Effective music therapy depends on dynamic matching among an individual’s emotional state, music emotion attributes, and intervention feedback. Existing intelligent music therapy studies have focused mainly on system implementation or recommendation performance, while paying insufficient attention to how emotion recognition informs therapeutic decision-making, how music features map onto regulation goals, and how user feedback revises subsequent interventions. Based on the valence-arousal emotion model, this study develops a personalized music therapy framework driven by multimodal emotion recognition. The framework maps the user’s current emotional state, target therapeutic state, and music emotion features into a unified space, and models the relationships among current-state congruence, target-state approach, user preference, and historical feedback through a therapeutic utility function. In implementation, a cross-modal Transformer is employed to fuse facial, speech, and physiological signals for user emotion representation; VGGish with dual-task learning is used to extract music valence and arousal features; and a hybrid strategy integrating content matching, collaborative filtering, and DQN-based feedback optimization is used to generate personalized music intervention plans. Experimental results show that the proposed method achieved 90.1% emotion-recognition accuracy and an F1 score of 89.6%. For music emotion prediction, the valence and arousal MAEs were 0.096 and 0.101, respectively, with a correlation coefficient of 0.861. The recommendation module achieved HR@10 and NDCG@10 values of 84.2% and 80.5%, respectively. In the short-term affective evaluation, the mean reduction in distance from the current state to the predefined target state was 0.214 for the proposed estimator, compared with 0.186 for Tensor Fusion. These results support the computational feasibility and adaptability of the framework, while not constituting evidence of clinical therapeutic efficacy.

Authors

Publication Details

Journal
Scientific Reports
Published
2026-11-21
DOI
https://doi.org/10.1038/s41598-026-77374-z
Primary Topic
Emotion and Mood Recognition
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multimodal emotion recognition and feedback adaptation for personalized music therapy

Kejing Chen
Scientific Reports
Emotion and Mood Recognition
article

Multimodal emotion recognition and feedback adaptation for personalized music therapy

Kejing Chen
article en

Abstract

Effective music therapy depends on dynamic matching among an individual’s emotional state, music emotion attributes, and intervention feedback. Existing intelligent music therapy studies have focused mainly on system implementation or recommendation performance, while paying insufficient attention to how emotion recognition informs therapeutic decision-making, how music features map onto regulation goals, and how user feedback revises subsequent interventions. Based on the valence-arousal emotion model, this study develops a personalized music therapy framework driven by multimodal emotion recognition. The framework maps the user’s current emotional state, target therapeutic state, and music emotion features into a unified space, and models the relationships among current-state congruence, target-state approach, user preference, and historical feedback through a therapeutic utility function. In implementation, a cross-modal Transformer is employed to fuse facial, speech, and physiological signals for user emotion representation; VGGish with dual-task learning is used to extract music valence and arousal features; and a hybrid strategy integrating content matching, collaborative filtering, and DQN-based feedback optimization is used to generate personalized music intervention plans. Experimental results show that the proposed method achieved 90.1% emotion-recognition accuracy and an F1 score of 89.6%. For music emotion prediction, the valence and arousal MAEs were 0.096 and 0.101, respectively, with a correlation coefficient of 0.861. The recommendation module achieved HR@10 and NDCG@10 values of 84.2% and 80.5%, respectively. In the short-term affective evaluation, the mean reduction in distance from the current state to the predefined target state was 0.214 for the proposed estimator, compared with 0.186 for Tensor Fusion. These results support the computational feasibility and adaptability of the framework, while not constituting evidence of clinical therapeutic efficacy.

Scientific Reports
Emotion and Mood Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon