Label-masking audits of health-record large language model agents pass answers that raise suspicion of drug misuse among recipients who know condition rates: a preregistered synthetic-cohort study
Preprint (author's original version); not peer reviewed. Objective: To measure when audits that mask a sensitive label pass large language model (LLM) agent answers from which recipients infer the attribute, by what auditor and recipient know. Materials and Methods: In a synthetic cohort (22,324 persons; drug misuse 15.5%), an LLM agent answered an employer's yearly questions after one of three audits: label masking alone, or with coarsening of literature-named conditions or of the three conditions most correlated with drug misuse. Recipients were given age and sex, the overall rate, or rates by condition. Leakage, the recipient's gain in balanced accuracy when flagging probabilities above 0.5, was measured by an LLM reading the answers (80 persons) and a classifier on the audited record (all persons). Hypotheses were preregistered. Results: Under label masking, the recipient given rates by condition flagged 42% with drug misuse and 10% without (leakage 0.158; 95% CI 0.075–0.240; adjusted p = 0.0042), although its area-under-the-curve gain was not distinguishable from zero. At the generator's correlation strength, merging the three correlated conditions into one name did not reduce classifier leakage by the preregistered 0.005 (reduction 0.0049), leaving 0.151 (0.142–0.162); LLM-measured leakage fell by 0.100 (0.037–0.165; no decision rule). Discussion: Label masking did not prevent inference by recipients given condition rates; merging correlated conditions had different effects in the two measurements. Conclusion: Those designing approval of health-record agents should name the assumed recipient, test audits against the strongest rate knowledge it could hold, and weigh leakage per attribute and measurement against coarsening's cost. Preregistration and addendum: doi:10.17605/OSF.IO/S5M9T. Code and data: doi:10.5281/zenodo.24338451 and https://github.com/mikotomiura/health-twin-sharing. The PDF contains the manuscript, Figures 1–4 and the Supplementary material. Version 2 (19 November 2026): the use of generative AI is stated in full (tools and versions, uses, and the authors' role) in the Acknowledgments; tables are double-spaced. Text, numbers, tables and figures are otherwise unchanged from version 1.
Authors
- Yuusuke Harada (ORCID: https://orcid.org/0009-0006-7252-5115)
- Mikto Miura (ORCID: https://orcid.org/0009-0000-4196-0508)
Institutions
- Hiroshima University (JP)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24344575
- Primary Topic
- Privacy-Preserving Technologies in Data
- Type
- preprint