Deciding How to Augment Language Models: Comparing Retrieval Strategies for Safe Oncology Decision Support

Small language models that run locally on hospital computers and search medical databases are promising tools for helping doctors. However, planning safe cancer care requires connecting many parts of a patient’s chart: everyday medications, organ function over time, past side effects, and inherited genes that affect how drugs work. It remains unclear whether small local models know how to connect these details and search for the right warnings to avoid hidden dangers. To address this question, this study introduces a benchmark of 150 composite oncology patients assembled from public registries across 21 cancer types and evaluated across an initial and a follow-up visit, with a subset of patients injected with plausible, database-traced clinical hazards. Eleven local open-weight models (2B to 31B parameters) and two frontier Gemini models were evaluated across three retrieval regimes: zero-shot baseline, upfront retrieval-augmented generation (RAG), and agentic tool use. In pre-specified safety parity tests, Gemma 4 31B equipped with agentic tools matched the safety of zero-shot Gemini 4 Argon, whereas smaller local models failed to close the safety gap. Upfront RAG reduced unsafe recommendations across local models by 15.1 percentage points (95% CI, 11.6 to 18.6), but caused excessive caution on safe cases. In contrast, agentic tool use reduced unsafe plans by only 6.4 percentage points; although effective for resistance mutations and follow-up adjustments, tool-directed models queried everyday medications in only 39% of drug-processing gene (pharmacogenomic) hazards, producing unsafe plans in 78% of unqueried cases. A deterministic post-generation plan check that reviews a finished treatment draft against the medical databases fixed roughly 60% of unsafe plans; when paired with agentic tool use, it matched the safety of upfront RAG (10.6% versus 10.5% unsafe) without making the model overly cautious on safe patients. Until small models get better at deciding what to search for on their own, using them safely requires pairing tool use with a deterministic final safety check.