Clinical scenarios in paediatric pulmonology: Can large language models fare better than trainee doctors?

<bold>Background:</bold> Generative large language models (LLMs) have shown immense potential in healthcare. There is speculation whether they can augment or even replace clinicians. <bold>Aims:</bold> We compared performance of 3 LLMs (ChatGPT, Bing and Bard) against early-career trainee doctors (TDs) in providing responses to complex paediatric respiratory scenarios. <bold>Methods:</bold> Six paediatric respiratory clinical scenarios (figure) were posed to TDs and the 3 LLMs (respondents). TDs were given an hour time limit with internet access (excluding LLMs). Responses were randomised and scored overall and on criteria (correctness, comprehensiveness, utility, plausibility, coherence and “humanness”) by six experts. Differences in scores between respondents and pairwise differences were tested for statistical significance. <bold>Results:</bold> ChatGPT (median overall score 7, IQR 6-8·25) outperformed Bard (median 6, IQR 5-7), Bing (median 4, IQR 3-5) and TDs (median 4, IQR 3-6) in all domains (p<0·001). ChatGPT outperformed TDs in all criteria (p<0·001). Bard scored better than TDs in coherence (p<0·05), but Bing and TDs scored similarly. Experts reliably identified Bing and Bard (but not ChatGPT) responses as non-human. No fabricated facts were present in LLM responses. <bold>Conclusions:</bold> ChatGPT outperformed TDs in providing clinically useful responses. It can augment expert decision making. This sets stage for future development of LLMs in clinical settings. <fig><object-id>erj;64/suppl_68/OA2762/F1</object-id><object-id>F1</object-id><object-id>F1</object-id><graphic></graphic></fig>