Cantata: A Multi-Head Architecture for Structured Long-Form Music Generation

We present Cantata, a single-stage codec language model for full-length lyrics-to-song generation. Cantata unifies two ideas: the long-context modeling of Llama-style backbones, which sustains coherence across an entire song, and the stacked multi-codebook decoding of neural-codec models, which packs a stream's parallel residual codebooks into one token sequence, a principle we generalize beyond a single stream. Within one model, it generates the vocal and accompaniment streams jointly, predicting every stream and codebook in parallel at each decoding step. A structured sequence format lays out the whole song in advance: all segment labels (intro, verse, chorus, bridge) and a track description precede generation, and each segment is then produced in turn, conditioned on its lyrics and all previously generated audio. To interleave the two streams' codebooks into one sequence, we compare several arrangements and adopt accompaniment-first ordering, which yields the lowest training loss. A shared codebook embedding with a lightweight per-stream low-rank residual (in the style of LoRA) makes the design efficiently extensible to more than two streams: a new stem, an instrument, backing vocals, or sound effects, reuses the shared embedding through a small residual rather than a full set of tables. We validate the architecture by training a 3B-parameter model, demonstrating that its many stream-codebook heads optimize stably together and that its loss holds steady across a song's successive segments.