Prism Tuning: Repulsion-Trained Seed Embeddings in a Frozen Transformer for Non-Redundant Generation
Large language models used to synthesize training data repeat themselves: the same fact or idea reappears under different wording, so the information content of a synthetic corpus grows far more slowly than its size. The prevailing remedy is post-hoc: generate a large pool, then discard near-duplicates. This pays to produce data that is thrown away and can only remove redundancy, never create coverage. We introduce Prism Tuning, an architecture that moves diversity to training time. A single frozen transformer is augmented with K small learnable "seed" vectors added to its input embeddings; the seeds define K parallel generation streams that run in one batched forward pass. A repulsion objective on the streams' output representations, back-propagated only into the seeds, drives the streams to occupy distinct semantic regions. Because the backbone is frozen and the seeds total a few kilobytes, diversity is amortized into parameters once and incurs zero inference-time coordination overhead. We give the complete formulation, including (i) the non-differentiability of decoded text and its resolution via a generated-position hidden-state surrogate, (ii) an anisotropy correction without which the repulsion signal is flat, and (iii) a structural style/content disentanglement of the seeds that blocks the degenerate "change tone, not content" solution. This paper presents the architecture, its formulation, and its integration cost; systematic empirical evaluation is left to future work.
Authors
- Gruhesh Sri Sai Karthik Kurra
Institutions
- Imperial College London (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24345494
- Primary Topic
- Topic Modeling
- Type
- preprint