Continuous-Space Diffusion and Flow Language Models: A Survey

Continuous-space diffusion and flow language models generate text by updating real-valued vectors (such as token embeddings, compressed codes, or sentence vectors) and decoding them into discrete words at readout. We organize this literature by three coupled choices: the continuous state being refined, the transition operator that updates it, and the text interface that converts states into tokens. Because language is discrete while the state space is continuous, adapting image diffusion to text encounters rounding ambiguity and embedding collapse under squared-error regression, uninformative noise schedules, distribution shift between clean-text codecs and learned priors, and mode averaging in few-step sampling. Across published comparisons with autoregressive and discrete-diffusion baselines, continuous models show no general advantage over same-scale autoregressive models on open-ended text; the closest prompt-continuation tie uses different training sequence lengths and separately tuned samplers. RePlaid matches or improves upon masked diffusion in likelihood bounds past compute-optimal budgets, and large-scale blockwise adaptation brings continuous models close to masked baselines on mathematical reasoning and code; however, continuous models still trail autoregressive baselines in held-out likelihood bounds, and scaling fits indicate that matching exact autoregressive negative log-likelihood requires substantially more training compute. Their clearest gains over autoregressive decoding appear in gradient-guided control, plan-level diversity, and task-specific reasoning after targeted fine-tuning. Under few-step budgets, trajectory-consistent distillation and stochastic bridges prevent entropy collapse, yet no standalone continuous generator has shown an end-to-end latency win over optimized autoregressive decoding at matched quality; the reported matched-quality latency gains rely on autoregressive verification of continuous drafts. We close with open problems in likelihood scaling, readout fidelity, few-step serving, representation reuse, and post-training.

Authors

Publication Details

Journal
Preprints.org
Published
2026-11-13
DOI
https://doi.org/10.20944/preprints202611.1395.v1
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Continuous-Space Diffusion and Flow Language Models: A Survey

Haocheng Mei, Yunhe Wang, Allen Chen, Chao Xu et al.
Preprints.org
Topic Modeling
preprint

Continuous-Space Diffusion and Flow Language Models: A Survey

Haocheng Mei, Yunhe Wang, Allen Chen, Chao Xu, Ye Yuan, Haojun Jiang, Yuchuan Tian, Wei He
preprint en

Abstract

Continuous-space diffusion and flow language models generate text by updating real-valued vectors (such as token embeddings, compressed codes, or sentence vectors) and decoding them into discrete words at readout. We organize this literature by three coupled choices: the continuous state being refined, the transition operator that updates it, and the text interface that converts states into tokens. Because language is discrete while the state space is continuous, adapting image diffusion to text encounters rounding ambiguity and embedding collapse under squared-error regression, uninformative noise schedules, distribution shift between clean-text codecs and learned priors, and mode averaging in few-step sampling. Across published comparisons with autoregressive and discrete-diffusion baselines, continuous models show no general advantage over same-scale autoregressive models on open-ended text; the closest prompt-continuation tie uses different training sequence lengths and separately tuned samplers. RePlaid matches or improves upon masked diffusion in likelihood bounds past compute-optimal budgets, and large-scale blockwise adaptation brings continuous models close to masked baselines on mathematical reasoning and code; however, continuous models still trail autoregressive baselines in held-out likelihood bounds, and scaling fits indicate that matching exact autoregressive negative log-likelihood requires substantially more training compute. Their clearest gains over autoregressive decoding appear in gradient-guided control, plan-level diversity, and task-specific reasoning after targeted fine-tuning. Under few-step budgets, trajectory-consistent distillation and stochastic bridges prevent entropy collapse, yet no standalone continuous generator has shown an end-to-end latency win over optimized autoregressive decoding at matched quality; the reported matched-quality latency gains rely on autoregressive verification of continuous drafts. We close with open problems in likelihood scaling, readout fidelity, few-step serving, representation reuse, and post-training.

Preprints.org
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon