A Fail-Safe Layered Guardrail for Emergency Detection in LLM-Based Health Assistants

Large language model (LLM) assistants that answer health questions face an asymmetric error cost: failing to recognise a medical emergency can cause serious harm, whereas an unnecessary escalation costs only inconvenience. We present the safety layer of HealthSense AI, a multi-agent health information assistant, and describe a layered guardrail in which a deterministic keyword screen routes each input, an LLM classifier makes the final decision on every input, and the deterministic layer serves as a fail-safe floor whenever the classifier cannot run. A second, independent guardrail screens generated output. We evaluate the design on 87 cases across two test rounds: a 56-case red-team suite written alongside the guardrail, and a 31-case independent set written from how people actually describe emergencies — lay somatic language, apologetic hedging, non-native English. The final system achieves 100% emergency recall (46/46), zero false alarms on 28 safe inputs and outputs, and 100% recall on 13 unsafe outputs. We also report two failures found during development: a keyword whitelist that scored perfectly on the suite it was built against but wrongly blocked 6 of 10 safe questions on the independent set, and a silent regression in which the LLM classifier was disabled without raising an error. Both shaped the final design. All code and test cases are public.