HiP-ETTS: Multimodal Emotional Text-to-Speech via Shared-Specific Emotion Modeling and Hierarchical Prior Projection and Refinement

Abstract Multimodal emotional prompts have broadened the controllability of emotional text-to-speech synthesis by allowing target emotions to be specified through diverse cues, including speech, text, and facial images. Nevertheless, existing methods still face two fundamental challenges: entangled emotional representations and the trade-off between content fidelity and emotional expressiveness. To address these challenges, we propose HiP-ETTS. It first decomposes emotional prompts into two fine-grained factors: an emotion-shared factor and a complementary emotion factor, which respectively represent VAD-related affective structure and complementary expressive information beyond the VAD-conditioned prediction. Conditioned on these emotion factors, HiP-ETTS then constructs a content-stable hierarchical emotional prior and adaptively projects it into an intermediate flow state for subsequent acoustic refinement, thereby enhancing emotional expressiveness while preserving linguistic content. Experiments across different emotional prompt conditions demonstrate that HiP-ETTS achieves lower or comparable word error rates and consistently higher emotional consistency scores than previous methods, while requiring fewer parameters and enabling faster inference.