Artificial Intelligence Chatbot as a Companion for Cancer Patients About Most Common Questions: Analysis of Readability and Quality

Aim: Advances on artificial intelligence (AI) have led to development of AI chatbots and more people are using AI chatbots to seek answers to their questions every day. We conducted this study to investigate the readability and quality of answers generated by large language model AI chatbots as companions in answering questions for cancer patients. Methods: After surveying 508 patients admitted to the outpatient clinic of Ankara University Faculty of Medicine, Department of Medical Oncology, we selected the most frequently asked questions about the four most common cancer types and general cancer knowledge. We asked these questions of ChatGPT (an AI chatbot from OpenAI) and calculated readability and quality scores, and the statistical difference between suggested and calculated reading scores. Means and the t-tests (one-way and/or paired) were used for statistical analysis. Results: A total of 57 questions, including those about colorectal, breast, lung, prostate cancer, and general cancer questions, were selected for analysis. The mean Flesch Reading Ease Score for all questions was 48.18 [standard deviation (SD) ±11.65], which was significantly lower than the suggested reading score of 60 points (p<0.01). The mean score for graded readability scores was 13.21 (SD ±2.49), which was consistent with college-level readability and significantly higher than a suggested value of 6th graders (p<0.01). The mean DISCERN score of all questions was 51.98 (SD ±7.27) and the Global Quality Score was 3.91 (SD ±0.69). Breast cancer responses were easier to read on graded scales (p=0.02) and had higher quality (p=0.05). Conclusion: ChatGPT may be a good companion for cancer patients despite its limitations, but it should be used carefully.