Accounting for Insulin-on-board in Large Language Model Advice about Pediatric Hyperglycemia

Large language models (LLMs) can be used by patients and caregivers to answer medical questions and the characterization of safety concerns in LLM outputs for patient-facing applications is an active area of research. In this work, we study LLM responses to caregiver questions about hyperglycemia in pediatric type 1 diabetes. We measure the frequency of a potential safety failure: advice to dose insulin in the absence of a clear warning to account for insulin-on-board. We found occurrences of the potential safety failure across four LLMs tested through programmatic endpoints and two LLMs tested through chat interfaces. The rates of the potential safety failure varied when different context was given about the patient's meal and insulin history and treatment regimen. These results highlight a need for guardrails for patients with diabetes who use LLMs to manage their care.