Triage Advice From General-Purpose Language Models in Health and Everyday Situations

Importance: One adult in 10 has consulted a chatbot for health information within 6 months, and nearly half of users do so to decide whether they need care. Referral regardless of need could add demand on stretched services. Objective: To compare referral to a professional by general-purpose language models in health and non-health situations against the level of contact that human raters judged necessary. Design, Setting, and Participants: Cross-sectional study of 4,346 responses generated in September 2026 by 53 general-purpose model configurations, released between March 2023 and September 2026 and accessed through APIs or run locally, to 82 investigator-written messages (41 health and 41 non-health). No patients were involved. Exposures: Health vs non-health scenario. Main Outcomes and Measures: Professional referral, unconditional or conditional on a future change or on doubt, coded by 3 language models and compared with the level of contact that physicians, pharmacists and 2 technically trained raters judged necessary. Results: Referral occurred in 74.7% of health and 46.3% of non-health responses, more often for health in every configuration (standardized difference, 28.3 percentage points; 95% CI, 18.3-37.0). Where raters judged contact unnecessary, 64.5% of health vs 27.6% of non-health responses recommended a professional (odds ratio, 17.08; 95% CI, 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%), and unconditional referral was 18.1% vs 6.6% (odds ratio, 4.66; 95% CI, 1.43-15.19). Where an in-person assessment was judged necessary, referral was similar (89.6% vs 84.5%). In medicines and physical health, referral was not associated with rated severity (odds ratio per point, 1.16; 95% CI, 0.53-2.55), unlike outside health (3.86; 95% CI, 1.79-8.33). The referral gap was 20.5 percentage points among configurations released in 2023 and 32.2 and 32.3 in 2025 and 2026 (4.8 percentage points per year; 95% CI, 2.3-7.2). Conclusions and Relevance: General-purpose language models referred people to a professional where a visit was needed, in health as outside it, but in health also in 2 in 3 situations that physicians and pharmacists judged manageable without contact, a pattern stronger in newer models. Evaluations of these systems should include whether their advice distinguishes situations that can be deferred or managed at a distance, alongside the referral of serious cases.