General-purpose LLM versus knowledge-base-grounded AI for real-world medical questions: a paired evaluation

Abstract Large language models (LLMs) are increasingly used to answer clinical questions, but comparisons by clinicians of general-purpose and knowledge-base-grounded systems remain limited. We conducted a non-blinded, paired evaluation at the 2026 congress of the German Society of Internal Medicine. Sixty-four physicians and medical students each assessed paired responses from LiSA, a knowledge-base-grounded clinical AI system developed by AMBOSS, and ChatGPT to one of 15 real-world internal medicine questions. Participants rated perceived quality, trust, and safety concerns. LiSA received higher ratings for overall quality (3.96 ± 1.20 vs. 3.48 ± 0.93; p = 0.002) and trust (5.11 ± 1.73 vs. 3.67 ± 1.39; p < 0.001). Safety concerns were reported in 15.6% of LiSA evaluations and 26.6% of ChatGPT evaluations ( p = 0.143). Clarity ratings did not differ significantly between systems, whereas LiSA was rated higher for completeness, actionability, and perceived medical accuracy, suggesting that the perceived advantage extended beyond linguistic clarity. In post hoc analyses, differences in quality and trust were greater among current than non-current AMBOSS users. These findings reflect perceived rather than objective performance, are susceptible to bias from non-blinding, question selection, and prior familiarity, and do not isolate the effect of knowledge-base grounding. Prospective, blinded studies are needed to determine clinical implications.