Measuring chemical safety and refusal in language models

A language model can decline a chemistry question for several reasons. It may detect a safety concern, fail to identify the molecule, or lack the facts needed to answer. When it does answer, the response may still be wrong or incomplete. Chemical safety evaluations need to distinguish these outcomes before interpreting differences between models. We propose a framework for doing so, using established practices from cheminformatics: verified identities and reference answers, suitable controls and baselines, independent validation, and statistics that account for related molecules and repeated measurements. We examine chemical safety studies, including SynthRefusal within onepot-Bench 0, ChemSafetyBench, DualChem, and broader scientific safety benchmarks. Eight detailed audits and additional methods readings identify both effective controls and gaps in the evidence. Our reanalyses reproduce large effects of molecular identifiers in ChemIDBench and system prompts in XSTest. A calculation from DualChem shows how excluding tool failures changes the apparent benefit of web access. Because these studies measure different outcomes, we do not pool their results. We propose an experiment that holds legitimate chemical safety questions fixed while changing the identifier and the information supplied to the model. It measures refusal and answer quality together, with separate judgments of whether withholding is appropriate. Separate extensions would test unsafe compliance and, where relevant, practical execution. Each proposed control addresses a specific uncertainty in interpreting model behavior.