When Models Train on Their Own Homework : Dose-Response of Model Collapse in a Word-Level Transformer and the 75% Human-Text Line

When language models are trained on text written by earlier models, quality, diversity and factual reliability can decline across generations. This is called model collapse. This paper quantifies how quickly the erosion occurs in a small system, and identifies how much human written text each generation’s training data needs to keep the vocabulary and diversity close to the original. For this, A word-level nanoGPT-style transformer was trained from scratch on CPU for three generations. The training data was 13,920 tokens from the opening passages of Charles Dickens’s 1859 novel A Tale of Two Cities, and the result was evaluated against the validation data of 1,579 tokens from the following passages. Two arms were then trained recursively for the three generations. In the collapse arm, each generation trains only on the previous generation’s output. In the mitigated arm, each generation mixes fresh real text with a synthetic share s of that output. Two experiments cover nine values of s, five seeds each, 45 runs in all, and every result below is a mean with a 95% confidence interval. The collapse arm loses two-thirds of its rare words by generation 3 (retention fell from 0.652 to 0.208), and the vocabulary of generated text shrank by 59.6% (1,821 to 736 words). However, mixing in real data reduced the loss. At synthetic shares of 0.1 to 0.25, models kept 72-88% of their original rare words coverage, and no run produced a repetition loop. At s ≥ 0.3, 6 of 20 runs produced repetition loops. Vocabulary per set fell with synthetic share. Average metrics barely separated s = 0.25 from s = 0.3, but worst-case sets did, which is exactly why worst-case checking belongs in every collapse evaluation. Real data protects against collapse in proportion to its share. For this setup, keeping at least 75% human-written data per generation kept the model near the real-text floor. The thresholds should be read as indicative: one novel, one small model, five seeds.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24249750
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Models Train on Their Own Homework : Dose-Response of Model Collapse in a Word-Level Transformer and the 75% Human-Text Line

Misha Saraf
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

When Models Train on Their Own Homework : Dose-Response of Model Collapse in a Word-Level Transformer and the 75% Human-Text Line

Misha Saraf
preprint en

Abstract

When language models are trained on text written by earlier models, quality, diversity and factual reliability can decline across generations. This is called model collapse. This paper quantifies how quickly the erosion occurs in a small system, and identifies how much human written text each generation’s training data needs to keep the vocabulary and diversity close to the original. For this, A word-level nanoGPT-style transformer was trained from scratch on CPU for three generations. The training data was 13,920 tokens from the opening passages of Charles Dickens’s 1859 novel A Tale of Two Cities, and the result was evaluated against the validation data of 1,579 tokens from the following passages. Two arms were then trained recursively for the three generations. In the collapse arm, each generation trains only on the previous generation’s output. In the mitigated arm, each generation mixes fresh real text with a synthetic share s of that output. Two experiments cover nine values of s, five seeds each, 45 runs in all, and every result below is a mean with a 95% confidence interval. The collapse arm loses two-thirds of its rare words by generation 3 (retention fell from 0.652 to 0.208), and the vocabulary of generated text shrank by 59.6% (1,821 to 736 words). However, mixing in real data reduced the loss. At synthetic shares of 0.1 to 0.25, models kept 72-88% of their original rare words coverage, and no run produced a repetition loop. At s ≥ 0.3, 6 of 20 runs produced repetition loops. Vocabulary per set fell with synthetic share. Average metrics barely separated s = 0.25 from s = 0.3, but worst-case sets did, which is exactly why worst-case checking belongs in every collapse evaluation. Real data protects against collapse in proportion to its share. For this setup, keeping at least 75% human-written data per generation kept the model near the real-text floor. The thresholds should be read as indicative: one novel, one small model, five seeds.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon