Improvised Performance Following in Real Time for Automatic Accompaniment

Real-time performance tracking is the core component of automatic accompaniment systems. Previous methods typically require full information on the performance score (the human part) and also assume very limited improvisations off the score, otherwise, the tracker will get lost. In this paper, we aim to track a fully improvised performance for automatic accompaniment, in which the only reference is the accompaniment score (the machine part). The key idea is to incorporate semantic-level alignment to model the correspondence between the user performance and the accompaniment. Our system contains two real-time components: The first component is an accompaniment-conditioned quantization model using a Long Short-Term Memory (LSTM) layer. For each performance note onset, we first compute its rough projected score position using previously estimated performance-to-score mapping and then refine the mapping using the quantization results. The second component is a playback control system, which updates the performance-to-score mapping via the performance-quantized time pair (similar to the performance-score alignment pair in traditional automatic accompaniment systems). Experiments show the system achieves better tracking results on expressive piano performance than baselines, allowing score-free tracking on both texture and tempo variations. Video demos are accessible on our demo page<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.