Evaluating ChatGPT’s Effectiveness in Providing Medical Information for Pregnant Women with Rheumatic Diseases

Objective: The growing use of ChatGPT as a source of health information highlights the need to assess its accuracy and adequacy. This study evaluated the accuracy and adequacy of ChatGPT (version 3.5) in responding to frequently asked questions from pregnant women with rheumatic diseases in both Turkish and English, aiming to assess its potential as a reliable source of patient information across languages in rheumatology and maternal-fetal medicine. Material and Methods: A total of 36 questions related to pregnancy and rheumatic diseases were obtained from Google and divided into seven subgroups. Questions were posed to ChatGPT in both Turkish and English and responses were evaluated on a 4-point scale by a rheumatologist (Expert 1) and a perinatologist (Expert 2). Mann-Whitney U test was used for statistical analysis (p < 0.05 was considered significant). Results: ChatGPT’s English responses demonstrated a higher rate of accuracy and completeness compared to its Turkish responses. In English, 91.6% of answers were rated as correct, compared to 75.0% in Turkish. Expert 1 rated the average score for Turkish responses as 3.64 ± 0.54 and for English responses as 3.89 ± 0.31, a difference that was statistically significant (p = 0.023). Expert 2 rated Turkish responses with an average score of 3.83 ± 0.37 and English responses with an average score of 3.94 ± 0.23, with no statistically significant difference (p = 0.136). Conclusion: ChatGPT demonstrates promise as an accessible source of information for pregnant women with rheumatic disease, but has limitations in its non-English responses. This highlights the need for improvement in language-specific training of language models. Further research is recommended to explore the performance of ChatGPT across multiple languages and medical specialties.