Abstract 4147550: Evaluating the Clinical Reasoning of GPT-4, Grok, and Gemini in Different Fields of Cardiology

Introduction: The use of Large Language Models (LLMs) in clinical practice seems increasingly possible, although comparative studies are needed to achieve standardization. GPT-4, Grok and Gemini are three commonly used LLMs. While these LLMs may exhibit some level of clinical reasoning, their performance could differ between them and between areas of cardiology. Aim: To evaluate the clinical reasoning of GPT-4, Grok and Gemini in distinct fields of cardiology. Methods: A total of 12 cases were selected from the AHA Circulation Journal, representing six cardiology areas: General Cardiology (GC), Interventional Cardiology (IC), Cardiac Imaging (CI), Electrophysiology (EP), Heart Failure (HF) and Congenital Heart Disease (CHD). Each area included two cases. We only provided the case presentation to the LLMs, ensuring they were unaware of the diagnosis and management. Four questions were asked with the following prompts: 1. What are the diagnostic steps?, 2. What is the most likely diagnosis?, 3. What is the differential diagnosis?, 4. What is the appropriate management?. The responses were systematically evaluated with a digital rubric through an international collaboration of 12 cardiologists from various institutions. Each cardiologist evaluated two cases in their area of expertise and was aware of the diagnosis and management. One-way ANOVA was used for analysis. Results: GPT-4 had a slightly best overall performance (Table 1) and the highest score in GC and CHD; Grok excelled in determining the management and was superior in CI, HF and IC; Gemini had the highest performance in EP, although these findings were not statistically significant (Table 2). Conclusions: This is one of the first studies comparing the clinical reasoning of LLMs in cardiology. Our results propose that GPT-4 has the best overall performance and Grok is superior indicating the management. More studies comparing LLMs are required for further standardization of their use.