Would AI Take a Research Year? Pilot Study Evaluating the Reliability of ChatGPT in Advising Plastic Surgery Applicants on Research Years

Introduction: Large language models (LLMs) such as OpenAI's GPT-4o are increasingly used to summarize information and report trends in available data for medical education. For integrated plastic surgery, the utility of LLMs to recommend taking a research year has not been established. We aim to establish the reliability of ChatGPT reproducibility of research year recommendations for medical students applying to integrated plastic surgery. Methods: De-identified, self-reported integrated plastics applicant profiles in publicly available Google Sheets from 2022-2025 were assembled. Inputs provided to GPT-4o (three runs per profile) included Step 2 CK (Clinical Knowledge) score, AOA designation, and research productivity. Research-year status and match outcome were withheld. The model returned a binary recommendation to pursue a research year. Reproducibility was summarized as cross-run concordance. We compared model recommendations with applicants' actual research-year decisions. Results: matched. Across the three computed runs, there was a 98% concordance in recommendations. The LLM recommended a research year for 32.7% (18/55) of entries, whereas 45.5% (25/55) actually undertook one (p = 0.208). Agreement between model recommendations and applicant decisions was 41.8% (p = 0.28). Conclusion: ChatGPT demonstrated internal consistency, but its recommendations could not predict which students would take a research year en route to a successful residency match.