Streaming Piano Transcription Based on Consistent Onset and Offset Decoding With Sustain Pedal Detection

This paper describes a streaming audio-to-MIDI transcription method that can sequentially translate a piano recording into a sequence of note-on and note-off events. The sequence-to-sequence learning nature of this task may call for using a Transformer model, which has been used for offline transcription and could be extended for streaming transcription with a causal restriction of the attention mechanism. We assume that the decoder of this model suffers from the performance limitation. Although time-frequency features useful for onset detection are considerably different from those for offset detection, the single decoder is trained to output a mixed sequence of onset and offset events without guarantee of the correspondence between the onset and offset events of the same note. To overcome this limitation, we propose a streaming encoder-decoder model that uses a convolutional encoder aggregating local acoustic features, followed by an autoregressive transformer decoder detecting a variable number of onset events and another decoder detecting the offset events of the active pitches with validation of the sustain pedal at each time frame. Experiments using the MAESTRO dataset showed that the proposed streaming method performed comparably with or even better than the state-of-the-art offline methods while significantly reducing the computational cost.