Bread and Bicycles: What Happens to an Interaction Claim When the Second Detail Means Nothing

Audits of language models increasingly report that disadvantage compounds when two things are true of a person at once. Before such a claim can be sized, the instrument has to be shown to measure people rather than words. Alongside every audited pair we stack pairs of socially neutral details built the same way and in the same slots: cycling to work and baking bread, walking a dog and learning guitar. Their interaction should be zero. On four of five instruments it is not, on all three open models, and the mechanism is visible in the data: the effect of a clause depends on the clause beside it. Four failing instruments could be a fact about our implementation, so the same neutral comparison is run on published work with the original authors' own code, under a second preregistration that requires one of their numbers to be reproduced before anything is added. Two targets passed that gate, and in both, neutral details move the statistic well above zero. On four further targets we could not assemble everything a rerun needs, and the five artifacts such a rerun depends on are listed: the trained weights behind a learned metric, the exact input to a preprocessing tool, the runtime version used for generation, the grouping behind each published cell, and the frame a fine-tuned model expects. Neither rerun shows that a published claim is false, and neither is offered as one. Two preregistrations with their dated addenda, all data, all code and all offline validators are included; every number regenerates without a GPU, and Appendix A lists all 98 tests behind the central table.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-04
DOI
https://doi.org/10.5281/zenodo.23959679
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Bread and Bicycles: What Happens to an Interaction Claim When the Second Detail Means Nothing

Michael Podgortsev
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Bread and Bicycles: What Happens to an Interaction Claim When the Second Detail Means Nothing

Michael Podgortsev
preprint en

Abstract

Audits of language models increasingly report that disadvantage compounds when two things are true of a person at once. Before such a claim can be sized, the instrument has to be shown to measure people rather than words. Alongside every audited pair we stack pairs of socially neutral details built the same way and in the same slots: cycling to work and baking bread, walking a dog and learning guitar. Their interaction should be zero. On four of five instruments it is not, on all three open models, and the mechanism is visible in the data: the effect of a clause depends on the clause beside it. Four failing instruments could be a fact about our implementation, so the same neutral comparison is run on published work with the original authors' own code, under a second preregistration that requires one of their numbers to be reproduced before anything is added. Two targets passed that gate, and in both, neutral details move the statistic well above zero. On four further targets we could not assemble everything a rerun needs, and the five artifacts such a rerun depends on are listed: the trained weights behind a learned metric, the exact input to a preprocessing tool, the runtime version used for generation, the grouping behind each published cell, and the frame a fine-tuned model expects. Neither rerun shows that a published claim is false, and neither is offered as one. Two preregistrations with their dated addenda, all data, all code and all offline validators are included; every number regenerates without a GPU, and Appendix A lists all 98 tests behind the central table.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon