Measuring Repetitive Sequences: A Lempel-Ziv Compression-based Approach with Transition-Based Tokenization in Music Analysis

Repetition is a key element of music, providing structure, enhancing melodies, and fostering familiarity. However, its objective quantification toward advances in music information retrieval and generative AI remains challenging. Because most compression algorithms exploit repetition in a sequence, the more compactly a sequence can be compressed, the more repetition is likely to be present, and Lempel-Ziv compression has been a useful technique toward this analysis. However, the method of tokenizing symbolic music for this analysis can affect the output; often, repetition in music takes place when a sequence is transposed to an alternate key. Though the notes may be different in absolute position, this is still a musically significant repetition, yet it would be regarded as “new” by a note-based tokenization scheme. In this research, we present a new transition-based tokenization scheme, and show that this tokenization method, when paired with compressibility analysis, can better reflect repetitive structure. We apply this analysis to 12 musical compositions, assessing their compressibility using both the note-based and transition-based tokenizations, finding t hat compositions with prominent transposition repetitions show a more distinct difference in the effect of the introduced tokenization scheme on their LZ ratio.