Do Language Models Know Which Drug Is Unsafe? Three Measurement Pitfalls and a Severity Gradient

Medication-safety benchmarks report that language models often choose a drug the patient's risk rules out, but not whether the models lack the knowledge or fail to use it. Separating the two for open models (7B–72B), with a decision rule written before each experiment, we find that three common measurement choices can each mislead. The answer key: on 25 random RxSafeBench interaction cases, three auditor models from different providers name the key's unsafe drug in only 20–28%. Drug popularity: on drug pairs drawn from FDA labels, how often a drug is cited separates interacting from non-interacting pairs (AUROC 0.79) better than any open model. Answer format: moving a model between written probabilities and one-token scoring shifts its accuracy by up to fifteen points. Once these are controlled, the knowledge is graded by severity, also within popularity strata. In a pre-registered confirmation on 900 new pairs, every model separates major interactions better than moderate ones; open models reach AUROC 0.59–0.73 on major and 0.45–0.57 on moderate interactions, while a frontier commercial model reaches 0.95 and 0.72, so the gap does not shrink where errors are most dangerous. Use is distractible: shown the risk sentence for the unsafe drug only, 8–72B models gain 14–24 points; when every candidate carries a similar-looking warning, as real labels do, they gain 1–9 points. A simple retriever and a hazard-review prompt make open models more cautious, not more accurate. Code, pre-registrations and raw model outputs are included.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24355134
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Do Language Models Know Which Drug Is Unsafe? Three Measurement Pitfalls and a Severity Gradient

Sebastian Goin del Rio
2 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
preprint

Do Language Models Know Which Drug Is Unsafe? Three Measurement Pitfalls and a Severity Gradient

Sebastian Goin del Rio
preprint en

Abstract

Medication-safety benchmarks report that language models often choose a drug the patient's risk rules out, but not whether the models lack the knowledge or fail to use it. Separating the two for open models (7B–72B), with a decision rule written before each experiment, we find that three common measurement choices can each mislead. The answer key: on 25 random RxSafeBench interaction cases, three auditor models from different providers name the key's unsafe drug in only 20–28%. Drug popularity: on drug pairs drawn from FDA labels, how often a drug is cited separates interacting from non-interacting pairs (AUROC 0.79) better than any open model. Answer format: moving a model between written probabilities and one-token scoring shifts its accuracy by up to fifteen points. Once these are controlled, the knowledge is graded by severity, also within popularity strata. In a pre-registered confirmation on 900 new pairs, every model separates major interactions better than moderate ones; open models reach AUROC 0.59–0.73 on major and 0.45–0.57 on moderate interactions, while a frontier commercial model reaches 0.95 and 0.72, so the gap does not shrink where errors are most dangerous. Use is distractible: shown the risk sentence for the unsafe drug only, 8–72B models gain 14–24 points; when every candidate carries a similar-looking warning, as real labels do, they gain 1–9 points. A simple retriever and a hazard-review prompt make open models more cautious, not more accurate. Code, pre-registrations and raw model outputs are included.

Zenodo (CERN European Organization for Nuclear Research)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon