Carrying memory across a code-list revision breaks a precision promise: two public streams, a watch that notices, and a request for counterexamples

A system that answers only the records it can answer at a stated precision, and leaves the rest to people, makes a promise: of the answers it gives, at least this share will be right. We report one way that promise fails and how quickly the failure can be seen. On two public record streams whose code list was revised so that code numbers were rewritten (Chicago food inspection violations, 2018 food code; OSHA severe injury reports, OIICS 3.0), a keeper that carried its memory over the revision promised 0.94 and 0.87 precision for the first 1,000 records after it and realized 0.44 and 0.007. A keeper that started its memory at the revision kept its promise in every range of records in which it answered. A watch that compares the last 100 answers whose rightness came back against the target raised its first alarm 54 and 163 records after the revision, with no alarm before it, and none on thirteen other public streams without a revision. Two score-threshold tools that freeze their threshold before the revision answered 92-100% of the records afterwards with fewer than 10% right. Everything here is two streams, derived (not human-confirmed) settlements, and one setting; a public kit rebuilds every table from the public data with a published package. We ask for counterexamples: a stream where carrying memory over such a revision keeps the promise, or where the watch stays silent after one. Replication kit: https://github.com/iyulab/gil-replication

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-09
DOI
https://doi.org/10.5281/zenodo.24069103
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Carrying memory across a code-list revision breaks a precision promise: two public streams, a watch that notices, and a request for counterexamples

iyulab
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Carrying memory across a code-list revision breaks a precision promise: two public streams, a watch that notices, and a request for counterexamples

iyulab
preprint en

Abstract

A system that answers only the records it can answer at a stated precision, and leaves the rest to people, makes a promise: of the answers it gives, at least this share will be right. We report one way that promise fails and how quickly the failure can be seen. On two public record streams whose code list was revised so that code numbers were rewritten (Chicago food inspection violations, 2018 food code; OSHA severe injury reports, OIICS 3.0), a keeper that carried its memory over the revision promised 0.94 and 0.87 precision for the first 1,000 records after it and realized 0.44 and 0.007. A keeper that started its memory at the revision kept its promise in every range of records in which it answered. A watch that compares the last 100 answers whose rightness came back against the target raised its first alarm 54 and 163 records after the revision, with no alarm before it, and none on thirteen other public streams without a revision. Two score-threshold tools that freeze their threshold before the revision answered 92-100% of the records afterwards with fewer than 10% right. Everything here is two streams, derived (not human-confirmed) settlements, and one setting; a public kit rebuilds every table from the public data with a published package. We ask for counterexamples: a stream where carrying memory over such a revision keeps the promise, or where the watch stays silent after one. Replication kit: https://github.com/iyulab/gil-replication

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon