A lightweight natural language conditioning framework for controllable polyphonic music generation

Abstract Controllable symbolic music generation remains a challenging problem, as many generative models can compose musically plausible sequences but fail to consistently follow user-defined stylistic prompts over long durations. This paper presents a lightweight natural language conditioning framework for polyphonic music generation that converts free-form text prompts into structured control tokens representing genre, tempo, and note density. Unlike computationally intensive text-to-audio systems, the proposed approach uses an efficient rule-based translation layer that injects deterministic constraints directly into the generation context. To examine the impact of architecture on long-term prompt adherence, two autoregressive models were evaluated: a Long Short-Term Memory (LSTM) network and a decoder-only Transformer. Both models were pre-trained on 11,000 MIDI files and later adapted through transfer learning after vocabulary expansion to incorporate the proposed conditioning tokens. While the cross-entropy converged similarly during training, the two architectures produced structurally different outputs. During the test, LSTM exhibited temporal collapse, overproducing note density (47.3 notes/s) and implausible polyphony (86.4%). In contrast, the Transformer generated more balanced outputs with note density of 7.1 notes/s and polyphony of 28.1%, substantially closer to the human baseline. These results demonstrate that token-level loss alone is insufficient for evaluating controllable music generation quality. More importantly, the results indicate that the Transformer architecture, using self-attention, provides stronger preservation of global prompt semantics and long-range musical coherence than the recurrent LSTM baseline under the evaluated experimental settings.

Authors

Institutions

Publication Details

Journal
Discover Artificial Intelligence
Published
2026-11-17
DOI
https://doi.org/10.1007/s44163-026-02725-8
Primary Topic
Music Technology and Sound Studies
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A lightweight natural language conditioning framework for controllable polyphonic music generation

D. Suma, Animesh Mishra
1 links
Discover Artificial Intelligence
Music Technology and Sound Studies
article

A lightweight natural language conditioning framework for controllable polyphonic music generation

D. Suma, Animesh Mishra
article en

Abstract

Abstract Controllable symbolic music generation remains a challenging problem, as many generative models can compose musically plausible sequences but fail to consistently follow user-defined stylistic prompts over long durations. This paper presents a lightweight natural language conditioning framework for polyphonic music generation that converts free-form text prompts into structured control tokens representing genre, tempo, and note density. Unlike computationally intensive text-to-audio systems, the proposed approach uses an efficient rule-based translation layer that injects deterministic constraints directly into the generation context. To examine the impact of architecture on long-term prompt adherence, two autoregressive models were evaluated: a Long Short-Term Memory (LSTM) network and a decoder-only Transformer. Both models were pre-trained on 11,000 MIDI files and later adapted through transfer learning after vocabulary expansion to incorporate the proposed conditioning tokens. While the cross-entropy converged similarly during training, the two architectures produced structurally different outputs. During the test, LSTM exhibited temporal collapse, overproducing note density (47.3 notes/s) and implausible polyphony (86.4%). In contrast, the Transformer generated more balanced outputs with note density of 7.1 notes/s and polyphony of 28.1%, substantially closer to the human baseline. These results demonstrate that token-level loss alone is insufficient for evaluating controllable music generation quality. More importantly, the results indicate that the Transformer architecture, using self-attention, provides stronger preservation of global prompt semantics and long-range musical coherence than the recurrent LSTM baseline under the evaluated experimental settings.

Discover Artificial IntelligenceVol. 6(1)
Manipal Academy of Higher Education (IN)
Music Technology and Sound Studies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon