Analyzing singing voice expressivity: Focus on singing voice musical dynamics

Musical dynamics, a key expressive dimension of the singing voice, play a vital role in shaping phrasing and conveying the desired emotional impact. Despite their importance, their formalization and standardization remain limited. This work addresses these challenges by proposing methodologies to analyze and interpret dynamics from both audio and score-performance perspectives. Our approach includes: (1) Comparative Musical Dynamics Analysis – examining variations between audio performances, (2) Interpreting Musical Dynamics from Scores – analyzing curated real-world audio performances paired with scores featuring rich dynamics labels, and (3) Analyzing Listener Agreement on Perceived Dynamics – investigating the subjectivity of interpretation. To support these approaches, we curate diverse datasets, including a synthetic dataset for choral singing, score-performance datasets from performer and listener perspectives, and karaoke datasets for imitation-based dynamics analysis. Our findings reveal that while synthetic data enables controlled comparisons, real-world performances exhibit musical dynamics absent in synthetic renditions. Using Romantic-era Lieder scores, we semi-automatically curated score-performance pairs through state-of-the-art source separation and alignment techniques to train a dynamics prediction model. Collaborating with expert musicians, we annotated scores with synchronized dynamics labels and examined inter-annotator agreement using computational linguistics methods. Additionally, we developed a system to identify vocal dynamics automatically, employing structural segmentation and machine learning models trained on the Western classical Lieder corpus. A preliminary study on Hindustani music revealed dynamics variations at strong beat positions. Our findings emphasize the value of personalized models and highlight the importance of context-window size in dynamics prediction tasks.