Modelling the Flow of Emotion: A Hierarchical Attention Network for Vocal Trajectory Classification

Conventional Vocal biomarkers have quantifiable correlates of mental health, and talk is a convenient and noninvasive indicator of mental health. The majority of the existing systems, though, consider every voice recording as a single observation and generate one emotional label per utterance instead of monitoring the temporal dynamics in the affective state. A single-point method is unable to reflect the changes in mood direction that have the most clinical implications a days-long change in emotional valence is much more informative than a single reading, but existing architectures do not support this.. Emotional state is longitudinal in nature and its course rather than its value at any given moment captures the most dynamics of relevance to well-being. We tackle this by formulating emotion analysis as a sequence classification task and training a Hierarchical Attention Network (HAN) to classify the emotional trend of a speaker as Improving, Declining or Stable. This model is based on chronologically ordered sequences of voice recordings sampled out of the MSP-Podcast corpus v1.7, with a twolevel Bidirectional LSTM using attention performing at both the acoustic frame level and the voice-entry level. Each entry representation is enriched with contextual metadata such as time-of-day and ambient noise level that is concatenated with acoustic features.. The labels of the trajectory ground-truth are obtained based on the slope of linear regression of per-entry valence scores. Using a speaker-independent test partition, the HAN is 97.0% accurate and 0.97-weighted F1-score. A ablation study attests that both sets of hierarchical attention make significant contributions, and that the entire architecture performs better than a typical Bi-LSTM by 11.6 percentage points. This study provides a principled approach to longitudinal mood tracking based on voice that goes beyond snapshot classification and tracks the dynamics of emotion. The system can handle a 10-item sequence with a 198.6 ms response, which is reasonable to implement in close-to-real-time.