General-purpose and medically adapted large language models versus oncologist consensus for immunotherapy videos

Introduction Patients receiving immune checkpoint inhibitors depend on accurate education to recognize immune-related adverse events and to know when to contact their oncology team, and many rely on online video of highly variable quality. Expert review of such material is resource-intensive, and large language models (LLMs) are increasingly proposed to support preliminary screening; whether any model agrees with specialist judgment closely enough to do so safely, and how often models pass expert-inadequate material as adequate, is unknown. Methods In this cross-sectional content analysis (reported per STROBE), we evaluated anonymized transcripts of 90 English-language YouTube videos on cancer immunotherapy. Three medical oncologists and four LLMs — ChatGPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and the medically adapted MedGemma 27B — independently scored every transcript using DISCERN, the Global Quality Score (GQS), and the Patient Education Materials Assessment Tool (PEMAT) understandability and actionability domains, each model in three separate runs, with expert majority consensus as the reference standard. Results Agreement among oncologists was good to excellent, whereas agreement among models was limited. By expert-majority classification, 35 of 90 transcripts (38.9%) met at least three of the four adequacy criteria and 24 (26.7%) met all four. ChatGPT-5.5 deviated least from expert scores on every instrument and classified adequacy most accurately (sensitivity 94.3%, 95% CI 81.4–98.4; specificity 89.1%, 78.2–94.9; Cohen’s kappa 0.82, 0.69–0.93). Gemini and MedGemma scored systematically more leniently under the tested conditions and produced 43 and 26 false-positive adequacy classifications, respectively, passing material the oncologists judged inadequate; the medically adapted model did not improve agreement in this evaluation, and raw popularity metrics did not identify high-quality content. Discussion The transcript-based educational content of most sampled videos did not meet all four adequacy criteria. These findings support further evaluation of selected model–prompt combinations for preliminary review of patient-education material, but they do not establish an operating threshold for clinical use, and the leniency observed in two models means expert review cannot be removed from the process.