Simulated evaluation of large language model stepwise diagnostic reasoning with real-world chest pain encounters and Bayesian networks

In this simulated assessment, GPT-4o demonstrated diagnostic biases toward rare conditions and differed substantially from normative probabilistic models and physician practice patterns. These discrepancies could lead to unnecessary over-triage and resource utilization. Integrating LLMs within more rigorous probabilistic frameworks and calibrating them to realistic disease prevalences may be essential for effectively harnessing their potential as clinical decision-support tools.