Can artificial intelligence evaluate the quality of <scp>YouTube</scp> videos on erectile dysfunction?

Artificial intelligence (AI) refers to the capability of computer systems to carry out tasks that conventionally require human intelligence, such as visual perception and decision-making. Multiple AI platforms, including chatbots, utilise a combination of natural language processing (NLP) and machine learning techniques. While deep learning algorithms are often used in NLP to interpret vast amounts of text data, NLP encompasses a broader range of techniques beyond just deep learning [1]. Perhaps the most well-known chatbot is Conversational Generative Pre-Trained Transformer (ChatGPT) [2]. ChatGPT emulates and interprets human language with high precision and has been tested for a variety of broad purposes, such as rephrasing manuscripts into more readable patient text [3], answering urology board questions [4], drafting or editing manuscripts [2, 5], and interpreting quality of on-line medical information [6]. YouTube medical content has several videos sometimes disseminating incorrect or misleading information to a vast audience [7]. Given the overwhelming amount of health-related material on social platforms, it is unrealistic for medical professionals to individually scrutinise each video. As lay health consumers continue relying on on-line sources for medical information, there is concern about the quality of information people receive [7]. To our knowledge, no previous research has been conducted evaluating ChatGPT's proficiency in assessing medical video content quality. Thus, this study aimed to feed YouTube video transcripts into text for ChatGPT-4 and examine whether it produced comparable outcomes for quality assessment of video content compared to quality assessment by humans (two medical students, and one urologist). We hypothesised that ChatGPT's findings would mirror those of our prior investigation given its high performance across a multitude of tasks in various sectors. In our previous study, we searched ‘erectile dysfunction’ (ED) on YouTube on a cookie-free, cache-cleared, private browsing setting on Google Chrome, and the search results were based on YouTube's default settings of proposed relevance [8]. Videos were included if content was relevant to ED treatment. Only videos for patient education were selected. Videos were excluded if content was not relevant to ED treatment, had no accompanying audio, had narration and/or text in a language other than English, or demonstrated a surgical technique without an explanation for the general public. Videos >10 min in length were excluded and the first 50 videos that met the inclusion criteria were included. Two validated instruments, the Patient Education Materials and Assessment Tool (PEMAT) and a modified five-question DISCERN were used to assess the quality of information in the videos [8]. Collected scores from the three evaluators (two medical students, and one urologist) were also assessed to ensure there was no interrater variability. For the present study, the same YouTube videos were transcribed using the AI tool ‘Glasp’. These transcriptions were then fed into ChatGPT-4, which employed two standardised assessment tools: Modified DISCERN and PEMAT. Given that ChatGPT-4's analysis was confined to text derived from video transcripts, it lacked the capability to evaluate visual content. Consequently, we omitted criteria pertaining to visual elements—e.g., PEMAT Question 12, which inquires if the material utilises visual cues, and PEMAT Question 13, which assesses the legibility and user-friendliness of on-screen text. ChatGPT-4 assessed each video transcript three times and the three scores were also assessed to ensure there was no interrater variability. The scores from ChatGPT-4 and the evaluators in our previous study [8] were compared using a two-tailed Student's t-test. The intraclass correlation coefficient was 0.95 for DISCERN and 0.94 for PEMAT among humans, compared to 0.88 for DISCERN and 0.95 for PEMAT for ChatGPT. There were both similarities and differences in the evaluations between humans and ChatGPT (Table 1). When examining specific evaluation questions, some showed no statistically significant difference between the two evaluators. Specifically, for DISCERN's Question 1 and PEMAT's Questions 3, 5, 10, and 11, both human and ChatGPT-4 evaluations were aligned. However, differences were evident in other areas. DISCERN's Questions 4 and 5, which probe whether a video references additional information sources or highlights areas of uncertainty, showed significant differences between the evaluations. Similarly, PEMAT's Questions 4 and 8, which respectively assess the use of complex medical terms in the video and the segmentation of complex information into shorter sections, also demonstrated significant differences between human and ChatGPT-4 evaluations. We explored the role of AI chatbots in evaluating the quality of medical information available on YouTube. With a growing tendency among the public to seek health-related information on-line, the risk of encountering misinformation is significant [7, 8]. This concern is amplified by the bold and misleading claims made about treatments, highlighting the need for robust mechanisms to ascertain the reliability of such content. Our study delves into the capabilities of AI, particularly AI-driven chatbots in assessing the quality of on-line medical information videos. The findings indicate that while AI possesses considerable potential, its effectiveness in evaluating information quality requires further optimisation. In a previous study, we engaged ChatGPT in utilising the DISCERN instrument to evaluate the quality of on-line medical articles, subsequently comparing its performance to human assessments [6]. We discovered that ChatGPT's assessments aligned with human judgements in determining whether the articles set and achieved clear aims, used reliable sources, provided balanced or unbiased information, listed additional sources of information, and highlighted areas of uncertainty. However, in the present study, we found that ChatGPT was unable to assess whether the video transcripts listed additional sources of information, or highlighted areas of uncertainty. In addition, the inability to examine visual features of the video greatly limits the role for ChatGPT in assessing overall video quality. While ChatGPT has shown proficiency in natural language understanding and conversational tasks, it often struggles with accurately evaluating the reliability of sources. This limitation arises due to its training data, which lacks comprehensive and up-to-date information from verified academic and clinical sources. As a result, the model can sometimes provide outdated or contextually inappropriate responses when asked to identify the credibility of references [1, 2, 5, 9]. Pan et al. [10] conducted an evaluation of ED information provided by various AI chatbots, including ChatGPT, Perplexity, ChatSonic, and Microsoft Bing AI, using the PEMAT and DISCERN instruments. Their study also underscored the alignment in PEMAT's understandability and DISCERN's overall information quality. However, our study identified significant differences in the assessment of the use of complex medical terms and the inclusion of reliable sources and additional information, aspects where ChatGPT struggled. Conversely, Pan et al. [10] reported generally high-quality information from AI chatbots, but noted limitations in understandability, actionability, and readability due to the use of medical jargon and lack of visual aids. Pan et al. [10] further highlight the potential of AI in evaluating on-line medical content while also acknowledging the need for further optimisation to enhance the actionability and readability of AI-generated health information. The potential of AI is immense, yet it carries a substantial responsibility. Our study underscores the importance of continuous validation and expert oversight to ensure the accuracy of AI-generated output. As people increasingly turn to AI chatbots for information, the integrity of these platforms is crucial, particularly in an on-line environment rife with misinformation. However, ethical concerns such as algorithmic bias and the inadvertent spread of misinformation remain significant challenges in healthcare. Algorithmic bias can lead to unequal treatment recommendations, especially if the model is trained on unbalanced datasets. Therefore, it is essential that AI models are developed using verified and current medical data, and regularly audited for biases to mitigate any misleading information. Additionally, enabling AI to apply recognised quality assessment tools, like DISCERN, may assist the public to identify and disregard inaccurate information. Considering the relatively high interrater agreement observed with ChatGPT in both this study and in another study [6], there is evident potential for AI to effectively utilise complex assessment tools. This capability could play a crucial role in enhancing the reliability of information that AI systems provide to users seeking medical knowledge on-line while addressing ethical concerns surrounding the accuracy and fairness of these outputs. Our study is not without limitations. Firstly, ChatGPT assessed only transcripts of videos, as opposed to watching the videos directly. Additionally, we relied solely on standardised tools, DISCERN and PEMAT, for assessing content quality, recognising that other methods exist. Lastly, our restricted sample size and the focus on a single clinical condition may limit the generalisability of the findings to a wider population. However, our approach was exploratory, and as the first of its kind, it provides a new viewpoint on AI's utility in assessing the quality of on-line health information. Future research should aim to incorporate diverse datasets and multiple clinical scenarios to validate the broader applicability of the conclusions drawn. Our aspiration should be to harness AI's capability to emulate expert human evaluations, or at the very least, to flag exceptionally good or notably poor content for human review. Focusing on these refinements will set the stage for semi-automated solutions that can effectively assist in the rigorous assessment of health-related content quality. This work was supported by National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) grant UE5 DK137308, and National Institutes of Health Grant R01 DK130991 to Ranjith Ramasamy. The authors have no conflicts of interest to declare.