Detecting undisclosed model changes from local session logs: a method, and a worked example in which the operator was one day off

Users of AI coding assistants have no version string, no changelog of behavior, and no ability to roll back to a prior model. When the tool changes, the operator has only an impression, and an impression is unfalsifiable. This note describes a method for turning that impression into a measurement. Claude Code writes the model name, client version, and reasoning effort into local session logs that most users never open. A script walks the archive and reports, per session, the model served and the density of the assistant's own self-corrections. It is demonstrated on a single operator's archive of 64 sessions covering 2026-07-20 to 2026-08-26. The operator stated on 2026-07-31, from behavior alone and with no tooling, that the assistant had changed. The logs place the model boundary at 2026-07-30. The claim preceded the measurement by four weeks and was one day off. The method is the contribution. The worked example demonstrates that it functions, and it is reported with its limitations intact: the sample is unbalanced (5 sessions against 52), the workload differs across the boundary, an independent control signal does not confirm it, and the pattern and the test are drawn from the same data. It is a change-point observation with a date attached. It is not a quality finding, and quoting it as one is a misuse of it. Section 1 discloses the conflict of interest in full: the paper was written by the system it measures. Reproduction: node scripts/model-drift.mjs ~/.claude/projects/ . The script reads local files only and writes nothing. Any user of the same tool already has the archive. The measurement worth taking is the reader's own.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-08-26
DOI
https://doi.org/10.5281/zenodo.22117618
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Detecting undisclosed model changes from local session logs: a method, and a worked example in which the operator was one day off

Nathan Hall
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
article

Detecting undisclosed model changes from local session logs: a method, and a worked example in which the operator was one day off

Nathan Hall
article en

Abstract

Users of AI coding assistants have no version string, no changelog of behavior, and no ability to roll back to a prior model. When the tool changes, the operator has only an impression, and an impression is unfalsifiable. This note describes a method for turning that impression into a measurement. Claude Code writes the model name, client version, and reasoning effort into local session logs that most users never open. A script walks the archive and reports, per session, the model served and the density of the assistant's own self-corrections. It is demonstrated on a single operator's archive of 64 sessions covering 2026-07-20 to 2026-08-26. The operator stated on 2026-07-31, from behavior alone and with no tooling, that the assistant had changed. The logs place the model boundary at 2026-07-30. The claim preceded the measurement by four weeks and was one day off. The method is the contribution. The worked example demonstrates that it functions, and it is reported with its limitations intact: the sample is unbalanced (5 sessions against 52), the workload differs across the boundary, an independent control signal does not confirm it, and the pattern and the test are drawn from the same data. It is a change-point observation with a date attached. It is not a quality finding, and quoting it as one is a misuse of it. Section 1 discloses the conflict of interest in full: the paper was written by the system it measures. Reproduction: node scripts/model-drift.mjs ~/.claude/projects/ . The script reads local files only and writes nothing. Any user of the same tool already has the archive. The measurement worth taking is the reader's own.

Zenodo (CERN European Organization for Nuclear Research)
MediaTek (China) (CN)
Openalex Percentile: Top 7%
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.