Disentanglement of prosodic meaning: Toward a framework for the analysis of nonverbal information in speech

We propose a theoretical framework and a cost-effective automated method for the interpretation of prosodic messages (e.g., chunking of information, emphasis, conversation action, emotion). At the core of the proposal is a hierarchy of layered prosodic messages that co-occur within the same intonation unit (0.5 to 2 s long). Motivated by this hierarchy, a procedure for the differential detection of three such co-occurring nonverbal messages is then described. In way of implementation, we produce a variant model of the WHISPER automatic speech recognition system that flags intonation unit boundaries, intonation unit prototypes, and emphases therein. The procedure required us to alter WHISPER's token combinations and significantly adjust its prediction process. The variant model was tested on four datasets that contain spontaneous and read speech, and performs on a par with similar human annotation, and often better, using relatively modest training data. Several insights regarding this implementation, such as model size and encoding methods, are described as well. We believe that the proposed framework, coupled with the results of its application herein, can greatly improve the analysis of speech and language, integrating contextual information and speaker intentions into linguistic descriptions for a large array of purposes with modest means.