Assessing Information Provided by ChatGPT: Heart Failure Versus Patent Ductus Arteriosus

Introduction The study aims to provide insights regarding the effectiveness of ChatGPT in providing accurate, complete, and understandable information related to heart failure (HF) and patent ductus arteriosus (PDA). The study aimed to identify whether the abundance of online resources and patient inquiries leads to ChatGPT providing more accurate, complete, and understandable responses for HF compared to the less common PDA. Methodology Ten open-ended questions related to HF and 11 questions related to PDA were formed, and answers generated in ChatGPT 3.5 were assessed for accuracy via a 6-point Likert scale and for completeness via a 3-point Likert scale. Understandability was assessed using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). Data analysis and visualization was conducted using R Software version 4.3.1. Statistical significance between accuracy and completeness for both the conditions was determined using Spearman's R and Mann-Whitney U tests. Results Relevant answers were generated for all questions on the first attempt. For HF-related answers, the mean accuracy score was 5.4 (SD=0.7) and the mean completeness score was 2.3 (SD=0.7), with a median understandability of 83.75% (IQR=77.8%-94.5%). Spearman's correlation coefficient (rs) for HF was 0.44854 (p=0.19353). For PDA, the mean accuracy was 5.09 (SD=0.7), completeness was 2.27 (SD=0.5), and median understandability was 94.5% (IQR=88.9%-100%). Spearman's correlation coefficient for PDA was 0.21409 (p=0.52731). Furthermore, the Mann-Whitney U-test revealed no significant differences in accuracy (p=0.35758) or completeness (p=0.88866) between the two conditions. Conclusions While the accuracy and completeness of answers related to HF were slightly higher compared to those for PDA, the understandability of PDA responses was notably better. There is a need for refining artificial intelligence-driven language models for medical information dissemination.