Emotional inference as embedded algorithm in live performance: From recognition to dramaturgy

Abstract This article examines what happens when emotional-inference algorithms are embedded as compositional tools within live performance systems rather than deployed as external classifiers. Through analysis of the Transmodal Emotional Amplification Listener (TEAL), a real-time multimodal system developed by the author for two performance works, it demonstrates how embedding transforms algorithmic function from recognition to composition. TEAL employs nested analysis windows that track interval statistics across multiple time scales, assigns composer-defined emotional parameters to these patterns and fuses them with gestural energy data extracted via MediaPipe. The composite signal passes through a locally hosted language model that generates narrative text, the Synthetic Dream. A secondary interpreter reverses the emotional parameters into MIDI and the Synthetic Dream into visual prompts, closing a bidirectional compositional loop in which the system’s generated notes are played back acoustically on a Yamaha Disklavier grand piano alongside the performer. The article introduces emotional sovereignty, defined as the performer’s right to define the affective ground on which algorithms operate, as a design principle for embedded affective systems. Drawing on music emotion recognition, interactive systems research and cybernetic art, it argues that bespoke compositional embedding offers a viable alternative to universal emotion recognition.