Evaluating the Performance of ChatGPT-4o Oncology Expert in Comparison to Standard Medical Oncology Knowledge: A Focus on Treatment-Related Clinical Questions

Integrating artificial intelligence (AI) into oncology can revolutionize decision-making by providing accurate information. This study evaluates the performance of ChatGPT-4o (OpenAI, San Francisco, CA) Oncology Expert, in addressing open-ended clinical oncology questions. Thirty-seven treatment-related questions on solid organ tumors were selected from a hematology-oncology textbook. Responses from ChatGPT-4o Oncology Expert and the textbook were anonymized and independently evaluated by two medical oncologists using a structured scoring system focused on accuracy and clinical justification. Statistical analysis, including paired t-tests, was conducted to compare scores, and interrater reliability was assessed using Cohen's Kappa. Oncology Expert achieved a significantly higher average score of 7.83 compared to the textbook's 7.0 (p < 0.01). In 10 cases, Oncology Expert provided more accurate and updated answers, demonstrating its ability to integrate recent medical knowledge. In 26 cases, both sources provided equally relevant answers, but the Oncology Expert's responses were clearer and easier to understand. Cohen's Kappa indicated almost perfect agreement (κ = 0.93). Both sources included outdated information for bladder cancer treatment, underscoring the need for regular updates. ChatGPT-4o Oncology Expert shows significant potential as a clinical tool in oncology by offering precise, up-to-date, and user-friendly responses. It could transform oncology practice by enhancing decision-making efficiency, improving educational tools, and serving as a reliable adjunct to clinical workflows. However, its integration requires regular updates, expert validation, and a collaborative approach to ensure reliability and relevance in the rapidly evolving field of oncology.