Label-masking audits of health-record large language model agents pass answers that raise suspicion of drug misuse among recipients who know condition rates: a preregistered synthetic-cohort study

Preprint (author's original version); not peer reviewed. Objective: To measure when audits that mask a sensitive label pass large language model (LLM) agent answers from which recipients infer the attribute, by what auditor and recipient know. Materials and Methods: In a synthetic cohort (22,324 persons; drug misuse 15.5%), an LLM agent answered an employer's yearly questions after one of three audits: label masking alone, or with coarsening of literature-named conditions or of the three conditions most correlated with drug misuse. Recipients were given age and sex, the overall rate, or rates by condition. Leakage, the recipient's gain in balanced accuracy when flagging probabilities above 0.5, was measured by an LLM reading the answers (80 persons) and a classifier on the audited record (all persons). Hypotheses were preregistered. Results: Under label masking, the recipient given rates by condition flagged 42% with drug misuse and 10% without (leakage 0.158; 95% CI 0.075–0.240; adjusted p = 0.0042), although its area-under-the-curve gain was not distinguishable from zero. At the generator's correlation strength, merging the three correlated conditions into one name did not reduce classifier leakage by the preregistered 0.005 (reduction 0.0049), leaving 0.151 (0.142–0.162); LLM-measured leakage fell by 0.100 (0.037–0.165; no decision rule). Discussion: Label masking did not prevent inference by recipients given condition rates; merging correlated conditions had different effects in the two measurements. Conclusion: Those designing approval of health-record agents should name the assumed recipient, test audits against the strongest rate knowledge it could hold, and weigh leakage per attribute and measurement against coarsening's cost. Preregistration and addendum: doi:10.17605/OSF.IO/S5M9T. Code and data: doi:10.5281/zenodo.24338451 and https://github.com/mikotomiura/health-twin-sharing. The PDF contains the manuscript, Figures 1–4 and the Supplementary material. Version 2 (19 November 2026): the use of generative AI is stated in full (tools and versions, uses, and the authors' role) in the Acknowledgments; tables are double-spaced. Text, numbers, tables and figures are otherwise unchanged from version 1.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24344575
Primary Topic
Privacy-Preserving Technologies in Data
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Label-masking audits of health-record large language model agents pass answers that raise suspicion of drug misuse among recipients who know condition rates: a preregistered synthetic-cohort study

Yuusuke Harada, Mikto Miura
1 links
Zenodo (CERN European Organization for Nuclear Research)
Privacy-Preserving Technologies in Data
preprint

Label-masking audits of health-record large language model agents pass answers that raise suspicion of drug misuse among recipients who know condition rates: a preregistered synthetic-cohort study

Yuusuke Harada, Mikto Miura
preprint en

Abstract

Preprint (author's original version); not peer reviewed. Objective: To measure when audits that mask a sensitive label pass large language model (LLM) agent answers from which recipients infer the attribute, by what auditor and recipient know. Materials and Methods: In a synthetic cohort (22,324 persons; drug misuse 15.5%), an LLM agent answered an employer's yearly questions after one of three audits: label masking alone, or with coarsening of literature-named conditions or of the three conditions most correlated with drug misuse. Recipients were given age and sex, the overall rate, or rates by condition. Leakage, the recipient's gain in balanced accuracy when flagging probabilities above 0.5, was measured by an LLM reading the answers (80 persons) and a classifier on the audited record (all persons). Hypotheses were preregistered. Results: Under label masking, the recipient given rates by condition flagged 42% with drug misuse and 10% without (leakage 0.158; 95% CI 0.075–0.240; adjusted p = 0.0042), although its area-under-the-curve gain was not distinguishable from zero. At the generator's correlation strength, merging the three correlated conditions into one name did not reduce classifier leakage by the preregistered 0.005 (reduction 0.0049), leaving 0.151 (0.142–0.162); LLM-measured leakage fell by 0.100 (0.037–0.165; no decision rule). Discussion: Label masking did not prevent inference by recipients given condition rates; merging correlated conditions had different effects in the two measurements. Conclusion: Those designing approval of health-record agents should name the assumed recipient, test audits against the strongest rate knowledge it could hold, and weigh leakage per attribute and measurement against coarsening's cost. Preregistration and addendum: doi:10.17605/OSF.IO/S5M9T. Code and data: doi:10.5281/zenodo.24338451 and https://github.com/mikotomiura/health-twin-sharing. The PDF contains the manuscript, Figures 1–4 and the Supplementary material. Version 2 (19 November 2026): the use of generative AI is stated in full (tools and versions, uses, and the authors' role) in the Acknowledgments; tables are double-spaced. Text, numbers, tables and figures are otherwise unchanged from version 1.

Zenodo (CERN European Organization for Nuclear Research)
Hiroshima University (JP)
Good health and well-being
Privacy-Preserving Technologies in Data
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon