Single Human-AI Fit: One Attribute, Seven Channels, Two Directions

A system built for an average user imposes a cost on everyone who is not that user. For physical systems this cost is measurable and has a closed form. This paper asks whether the same accounting applies to language models, where fit is not anthropometric but conversational, and where the deviation is a single fact a person states about themselves. We run a counterfactual audit on three open models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3). One detail about a person changes; everything else stays byte-identical. Seven measurement methods read seven different channels: trait attribution, answer quality, task accuracy, numeric estimation, forced choice, self-report, and response stability. Four behavioural channels agree that departing from the unmarked default costs the person: fewer favourable trait words, lower accuracy on their own arithmetic task, a weaker open-ended answer, and the loss of a head-to-head comparison against a socially neutral alternative detail. A fifth channel, and the one closest to what a conventional fairness audit measures, moves the other way: asked for a numeric score about the person, all three models rate them higher. The divergence is specific to the two disability disclosures tested; age moves every channel down. We report the negative results in full. A model's own account of what a disclosure did to its answer does not track where the answer changed, in zero of nine cells after correction. Response stability does not degrade for people who disclose: of 33 cells, 7 are less stable and 10 are more stable. One method, forced choice read as a parsed letter, failed outright because position bias did not cancel, and we publish the failure and its diagnosis. Task-dependent bias is an established finding and we do not claim it. Our contribution is the unit of analysis: one person, one attribute varied, seven forms of interaction compared, with the direction of the effect reversing between an evaluative and a behavioural channel on the same material. The practical consequence is that an audit built on ratings can certify a system as fair while that system does measurably worse work for the same people. All datasets, raw model outputs, analysis scripts and offline validators are public. Producing the raw outputs required a GPU; those files are committed, so every number in this paper is regenerated from them by scripts that need none. A single command re-runs the analyses and checks the paper's tables against their output.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-05
DOI
https://doi.org/10.5281/zenodo.22364497
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Single Human-AI Fit: One Attribute, Seven Channels, Two Directions

Michael Podgortsev
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Single Human-AI Fit: One Attribute, Seven Channels, Two Directions

Michael Podgortsev
preprint en

Abstract

A system built for an average user imposes a cost on everyone who is not that user. For physical systems this cost is measurable and has a closed form. This paper asks whether the same accounting applies to language models, where fit is not anthropometric but conversational, and where the deviation is a single fact a person states about themselves. We run a counterfactual audit on three open models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3). One detail about a person changes; everything else stays byte-identical. Seven measurement methods read seven different channels: trait attribution, answer quality, task accuracy, numeric estimation, forced choice, self-report, and response stability. Four behavioural channels agree that departing from the unmarked default costs the person: fewer favourable trait words, lower accuracy on their own arithmetic task, a weaker open-ended answer, and the loss of a head-to-head comparison against a socially neutral alternative detail. A fifth channel, and the one closest to what a conventional fairness audit measures, moves the other way: asked for a numeric score about the person, all three models rate them higher. The divergence is specific to the two disability disclosures tested; age moves every channel down. We report the negative results in full. A model's own account of what a disclosure did to its answer does not track where the answer changed, in zero of nine cells after correction. Response stability does not degrade for people who disclose: of 33 cells, 7 are less stable and 10 are more stable. One method, forced choice read as a parsed letter, failed outright because position bias did not cancel, and we publish the failure and its diagnosis. Task-dependent bias is an established finding and we do not claim it. Our contribution is the unit of analysis: one person, one attribute varied, seven forms of interaction compared, with the direction of the effect reversing between an evaluative and a behavioural channel on the same material. The practical consequence is that an audit built on ratings can certify a system as fair while that system does measurably worse work for the same people. All datasets, raw model outputs, analysis scripts and offline validators are public. Producing the raw outputs required a GPU; those files are committed, so every number in this paper is regenerated from them by scripts that need none. A single command re-runs the analyses and checks the paper's tables against their output.

Zenodo (CERN European Organization for Nuclear Research)
Peace, Justice and strong institutions
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.