Crisis hand-over in staff-facing AI assistants: an independent benchmark of a small policy-conditioned guard and four open guard models

AI assistants used by staff in clinics, schools and public agencies must recognise when a message signals that someone may be at risk, and hand the conversation to a person. We built a benchmark to test this without our own authorship: 1,193 messages (58 expected crisis, 1,135 not) taken from three public safety datasets (Aegis 2.0, SimpleSafetyTests, XSTest), mapped to crisis or not-crisis by a rule fixed before any download, and frozen by SHA-256 before any system was scored. We scored a ~150-million-parameter classifier that reads an office's written guard policy together with the message, and four open guard models given the same policy wording: Llama Guard 3 (8B and 1B), ShieldGemma 9B and Granite Guardian 3 8B. Each system was run once, under a trained (clinic) and an unseen (school) policy. The classifier caught 105 of 116 crisis cases (90.5%); the other models caught 81.9–87.9%. In paired McNemar tests with Holm adjustment, no difference in crisis recall was significant. The benchmark has only 58 crisis messages, so these tests cannot separate rates this close. The classifier's false-crisis rate on safe messages was 31.0%, significantly higher than both Llama Guard models (3.8% and 13.6%) and ShieldGemma (1.1%), and not different from Granite Guardian (30.6%). Its errors concentrate on dark but safe topics (history, fiction, definitions) and on eating-disorder requests, which its policies did not name. We release the benchmark, protocol, per-message predictions and code. Content warning: the benchmark contains messages about suicide, self-harm and eating disorders, written by safety researchers to test AI systems.