Can ChatGPT Take My Call? evaluating AI in gynecologic oncology telephone triage

• ChatGPT matched fellow’s triage decisions for most patient calls. • When AI management differed, it favored patient safety with conservative over-triage. • Physicians rated AI-generated responses as highly accurate, clear, and useful. As access to artificial intelligence (AI) expands, patients and clinicians increasingly rely on these platforms for medical information and guidance. This study assessed ChatGPT’s ability to triage and respond to gynecologic oncology (GO) patient telephone calls. In this cross-sectional study, 30 patient scenarios received by on-call GO fellows were evaluated using ChatGPT-4o. Four physicians independently rated responses for accuracy, comprehensiveness, clarity, relevance, and applicability using a 5-point Likert scale (1 = very poor; 5 = very good) and assessed the presence of misinformation (yes/no). Wilcoxon signed-rank tests compared mean ratings with a predefined threshold of 3.0 (“acceptable”), with significance set at p < 0.05. The prevalence of misinformation was calculated with 95% confidence intervals (CI). Agreement with provider recommendations was assessed using Cohen’s kappa. Misclassification patterns (over-triage vs under-triage) were summarized. Among 30 scenarios, exact agreement between AI and fellows occurred in 26 cases (86.7%). Misinformation was identified in 4 responses (3.3%; 95% CI, 0.6–16.7%). Misclassifications included three cases (10.0%) over-triaged to clinic within one week and one case (3.3%) under-triaged to stay home rather than be seen within one week. In all 10 scenarios where the fellows’ recommended emergency department evaluation, AI issued the same recommendation. Agreement between AI and fellow triage decisions was high (κ = 0.87). Across all categories, physician ratings of AI responses significantly exceeded the acceptable threshold (p < 0.001). ChatGPT demonstrated high reliability in triaging and responding to GO patient telephone calls, producing clear, accurate, and clinically appropriate guidance. Its consistent performance in urgent scenarios and tendency toward conservative triage suggest that AI may serve as a valuable adjunct to support on-call personnel and enhance after-hours triage workflows in GO.