Performance of a Large Language Model on the Anesthesiology Continuing Education Exam

As of August 2023, more than 180 million people have used the large language model Chat Generative Pre-Trained Transformer (ChatGPT).1 There is emerging evidence that large language models can solve novel complex problems in a humanlike way.2 If proven accurate, large language models such as ChatGPT could facilitate real-time perioperative decision-making by giving anesthesiologists rapid access to evidence-based responses to clinical queries. ChatGPT performed “at or near the passing threshold” for the United States Medical Licensing Examination (USMLE) and displayed evidence of deductive reasoning.3 ChatGPT also performed well on board exam questions in various medical specialties, including neurosurgery and neonatology.4,5 While questions from the American Board of Anesthesiology (Raleigh, North Carolina) questions are not publicly available, previous researchers have used questions from a board examination preparation book and found that ChatGPT also performs satisfactorily on these anesthesiology questions.6 One notable concern of large language models is that they will occasionally “hallucinate” and use fabricated information to support their responses, which could lead to medical errors.7 The evidence of fabrications in large language model responses to anesthesiology questions is mixed.8,9 This exploratory analysis aimed to assess the performance of ChatGPT, including the likelihood of fabrication, using the 2023 Anesthesiology Continuing Education (ACE) test questions. These questions were designed to help practicing anesthesiologists “assess the current state of [their] knowledge of fundamental subspecialty anesthesia topics and identify knowledge gaps.”10 We assessed the performance of ChatGPT using four criteria (accuracy, fabrication, insight, and concordance) and examined the association between the performance of ChatGPT and question characteristics.The University of Rochester Research Study Review Board determined that this study was not human subjects research and thus did not require review and approval from the Board. This study used 100 multiple-choice questions from the annual ACE question bank, which is copyrighted by the American Society of Anesthesiologists (ASA; Schaumburg, Illinois) and was used with permission from ASA for this study. Each question was assigned a difficulty level reflecting the knowledge expected from a resident physician having completed the first year (basic [n = 39]), second year (intermediate [n = 35]), and third year of residency (advanced [n = 20]) and subspecialty area of expertise (general [n = 61], cardiac [n = 14], or other [n = 19]). This categorization was based on the consensus of three raters, all of whom were either board-certified anesthesiologists or anesthesiology residents, with 81% of the questions evaluated by at least two board-certified anesthesiologists. Kappa analysis demonstrated fair interrater agreement (κ = 0.38) for the question difficulty level. Each question was presented verbatim to ChatGPT 10 times to account for random variation in ChatGPT responses. Every question was presented five times with and five times without forced justification (“Explain the rationale for your choice”). Six of the 100 questions were excluded because they included media images that could not be uploaded into ChatGPT. Every iteration was performed in a new ChatGPT session to avoid memory-retention bias in the model. We assumed that Version 3.5 of ChatGPT, trained using data up to September 2021, did not have access to the 2023 ACE questions, ensuring that the questions presented to ChatGPT in this study represented a valid out-of-training sample. The accuracy of all answers was judged by comparing the ChatGPT answer (i.e., selection of either “A,” “B,” or “C”) to the published answers on the ACE exam. Explanations generated by ChatGPT (which were also provided by ChatGPT in most cases even when not requested) were each scored by the three independent raters (described earlier) using prespecified criteria based on a standardized rubric3 and the final scores were based on consensus (i.e., agreement by at least two raters). ChatGPT’s explanations were scored as having a fabrication if they contained any factually incorrect statements in the response. Explanations were graded as discordant if any part of the ChatGPT explanation contradicted itself. Explanations were graded as insightful if they (1) were nondefinitional (the explanation was not circular), (2) were deductional (required deduction or knowledge that is not included in the question), and (3) did not rely on false information to reach the answer (i.e., key statements in the explanation were factually correct).3 Multivariable logistic regression was used to examine the association between the large language model performance on each criterion and question characteristics (difficulty [basic, intermediate, advanced], expertise [general, cardiac, other]), and the use of forced justification. Cluster robust variance estimators were used to account for the clustering of observations within questions (i.e., repeated questions).Overall, 66.5% of responses were accurate. Accuracy was 65.9% without forced justification and 67.1% with forced justification (P = 0.70). Basic questions were answered with 84.4% accuracy compared to 52.6% for advanced questions (P < 0.001). After adjustment for specialization and prompting strategy, response accuracy was still associated with question difficulty (fig. 1). Compared to basic questions, responses to intermediate (adjusted odds ratio, 0.21; 95% CI, 0.08 to 0.55; P = 0.001) and advanced questions (adjusted odds ratio, 0.22; 95% CI, 0.07 to 0.69; P = 0.01) were less likely to be accurate. Compared to basic questions, explanations to intermediate (adjusted odds ratio, 3.44; 95% CI, 1.36 to 8.68; P = 0.009) and advanced questions (adjusted odds ratio, 5.35; 95% CI, 1.82 to 15.8; P = 0.002) were more likely to contain fabricated information. Explanations to advanced questions were less likely to demonstrate insight (adjusted odds ratio, 0.29; 95% CI, 0.10 to 0.83; P = 0.02) compared to basic questions (fig. 2). Kappa analysis revealed moderate interrater agreement across raters: fabrication (κ = 0.58), insight (κ = 0.57), and concordance (κ = 0.43).In this exploratory study, we found that the accuracy and likelihood of fabrications of ChatGPT responses to anesthesiology ACE questions were associated with the difficulty level of the questions. Fabricated information can be challenging to identify and could lead large language model users to wrongly accept false large language model-generated responses to medical queries as true. Responses to intermediate and advanced questions had 3.4- and 5.4-fold higher odds ratios, respectively, of including factually incorrect information compared to responses to basic questions. This study has several limitations. First, we used ChatGPT version 3.5 instead of 4.0, which likely led us to underestimate the performance of ChatGPT, including the likelihood of fabrications.8 We used ChatGPT version 3.5 to minimize the risk that ChatGPT would have access to the specific questions and answers included in the ASA 2023 ACE questions. Second, the questions in this study represent only a small subset of anesthesiology knowledge, thus limiting the generalizability of our findings. Third, the factual accuracy (i.e., fabrication) of each ChatGPT statement in the response was not manually verified using existing authoritative sources such as anesthesiology textbooks and not all raters were board-certified anesthesiologists. Fourth, there was no validated standard by which to grade the difficulty level of each question. Fifth, this study was limited to simple prompting, and did not explore more advanced prompting strategies such as chain-of-thought prompting, which is intended to improve large language model performance using an iterative approach that resembles human approaches to problem-solving. Sixth, we were unable to compare the performance of the large language model to that of anesthesiologists because this information is not made public by the ASA. Finally, this study was not designed to assess the likelihood of patient harm resulting from clinicians acting on inaccurate large language model responses. Future studies to assess the problem-solving skills of large language models that are better optimized for medical applications, using techniques such as prompt tuning,11 are necessary to better understand the potential usefulness of large language models to enhance perioperative decision-making and improve patient outcomes.This work was supported by the Department of Anesthesiology and Perioperative Medicine at the University of Rochester School of Medicine and Dentistry (Rochester, New York). Dr. Glance reports grant support from the National Institute on Aging (R01AG074492; Bethesda, Maryland) and the National Institute of Nursing Research (R01NR016865; Bethesda, Maryland).The authors declare no competing interests.