Sounding Like Which AI? Machine-Style Overlap of arXiv Abstracts with Language Models of Different Vintages

We ask not how much text was produced with LLM help but how much writing, by any author, falls on the LLM side of a stylistic boundary, and which LLMs define that side. We define Machine-Style Overlap (MSO): the share of documents that a calibrated classifier on style features that ignore content-word identity judges at least as likely to be LLM output as pre-ChatGPT human writing. In 131,448 arXiv abstracts (2015–2026), MSO was 9.2% before ChatGPT and by June 2025 stood 25.4 percentage points (pp) above its pre-ChatGPT trend, far above placebo gaps. Because recent LLMs learn from recent writing, we compare generators by vintage. Against generators whose pretraining data predate ChatGPT, the gap is smaller (7.5–13.1 pp by post-trend fit; 3.5–7.6 pp for two models run from titles). Single-generator overlap mostly rose then fell, peaking later for newer generators even at equal text length (exploratory rank correlation 0.79, n = 18; 0.66 without three added post hoc). A fixed reference set, like a fixed marker-word list (delve), can show the shift receding while overlap with newer models still rises. In US school essays, English learners' essays fall in the region less often, largely because of essay scores. AI use: the study was carried out by an AI system (Claude Opus 5.5, Anthropic) under the author's direction and is disclosed in full in the paper's Acknowledgements. The author read and approved this version. Code, data and results: https://doi.org/10.5281/zenodo.23343987

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-13
DOI
https://doi.org/10.5281/zenodo.23344296
Primary Topic
Authorship Attribution and Profiling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Sounding Like Which AI? Machine-Style Overlap of arXiv Abstracts with Language Models of Different Vintages

Natik Chhabra
Zenodo (CERN European Organization for Nuclear Research)
Authorship Attribution and Profiling
preprint

Sounding Like Which AI? Machine-Style Overlap of arXiv Abstracts with Language Models of Different Vintages

Natik Chhabra
preprint en

Abstract

We ask not how much text was produced with LLM help but how much writing, by any author, falls on the LLM side of a stylistic boundary, and which LLMs define that side. We define Machine-Style Overlap (MSO): the share of documents that a calibrated classifier on style features that ignore content-word identity judges at least as likely to be LLM output as pre-ChatGPT human writing. In 131,448 arXiv abstracts (2015–2026), MSO was 9.2% before ChatGPT and by June 2025 stood 25.4 percentage points (pp) above its pre-ChatGPT trend, far above placebo gaps. Because recent LLMs learn from recent writing, we compare generators by vintage. Against generators whose pretraining data predate ChatGPT, the gap is smaller (7.5–13.1 pp by post-trend fit; 3.5–7.6 pp for two models run from titles). Single-generator overlap mostly rose then fell, peaking later for newer generators even at equal text length (exploratory rank correlation 0.79, n = 18; 0.66 without three added post hoc). A fixed reference set, like a fixed marker-word list (delve), can show the shift receding while overlap with newer models still rises. In US school essays, English learners' essays fall in the region less often, largely because of essay scores. AI use: the study was carried out by an AI system (Claude Opus 5.5, Anthropic) under the author's direction and is disclosed in full in the paper's Acknowledgements. The author read and approved this version. Code, data and results: https://doi.org/10.5281/zenodo.23343987

Zenodo (CERN European Organization for Nuclear Research)
Authorship Attribution and Profiling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.