A lightweight natural language conditioning framework for controllable polyphonic music generation
Abstract Controllable symbolic music generation remains a challenging problem, as many generative models can compose musically plausible sequences but fail to consistently follow user-defined stylistic prompts over long durations. This paper presents a lightweight natural language conditioning framework for polyphonic music generation that converts free-form text prompts into structured control tokens representing genre, tempo, and note density. Unlike computationally intensive text-to-audio systems, the proposed approach uses an efficient rule-based translation layer that injects deterministic constraints directly into the generation context. To examine the impact of architecture on long-term prompt adherence, two autoregressive models were evaluated: a Long Short-Term Memory (LSTM) network and a decoder-only Transformer. Both models were pre-trained on 11,000 MIDI files and later adapted through transfer learning after vocabulary expansion to incorporate the proposed conditioning tokens. While the cross-entropy converged similarly during training, the two architectures produced structurally different outputs. During the test, LSTM exhibited temporal collapse, overproducing note density (47.3 notes/s) and implausible polyphony (86.4%). In contrast, the Transformer generated more balanced outputs with note density of 7.1 notes/s and polyphony of 28.1%, substantially closer to the human baseline. These results demonstrate that token-level loss alone is insufficient for evaluating controllable music generation quality. More importantly, the results indicate that the Transformer architecture, using self-attention, provides stronger preservation of global prompt semantics and long-range musical coherence than the recurrent LSTM baseline under the evaluated experimental settings.
Authors
- D. Suma (ORCID: https://orcid.org/0000-0003-2782-6702)
- Animesh Mishra
Institutions
- Manipal Academy of Higher Education (IN)
Publication Details
- Journal
- Discover Artificial Intelligence
- Published
- 2026-11-17
- DOI
- https://doi.org/10.1007/s44163-026-02725-8
- Primary Topic
- Music Technology and Sound Studies
- Type
- article
- Field-Weighted Citation Impact
- 0.00