Accuracy and efficacy of using generative AI for pathways review meeting summaries.

617 Background: The Pathways program at Dana Farber Cancer Institute (DFCI) holds regular disease-specific meetings to discuss proposed pathway changes, (e.g., new drug approvals and updates to clinical practice). The documentation of these meetings includes a summary that synthesizes the decisions made and the reasoning behind them. Historically, the disease center MD (DCMD) was responsible for manually compiling the summary, a process that was time-consuming and often delayed due to competing priorities. To improve this process, we explored the feasibility and benefits of using a generative AI tool to compile summaries of Pathways Review Meetings. Methods: From February to April 2025, the Pathways team utilized GPT4DFCI (a private, secure, HIPPA-compliant generative AI tool available at DFCI) to generate summaries of Pathways Review Meetings. We developed a customized prompt specifying the key details to include in the summary, such as “key trial details, the reasoning for adopting or not adopting each regimen into pathways, as well as a thorough exploration of the benefits, concerns, limitations, and cost considerations”. Review Meetings were recorded, and Zoom transcripts were produced. The transcripts were fed into GPT4DFCI using the customized prompt. The generated summaries were reviewed with the DCMD to ensure accuracy. GPT4DFCI was used to generate summaries for 8 Pathway Review Meetings, covering 26 agenda items. Results: Using GPT4DFCI reduced the median turnaround time for completing summaries to 2 business days, a significant improvement from the previous median of 10 business days. Among the 26 agenda item summaries generated by GPT4DFCI: 12 summaries (46%) required no edits, indicating the DCMD found them accurate and comprehensive; 14 summaries (54%) were edited to correct grammatical errors, misspelling of drug and study names, and sentence formatting. 2 of the summaries contained hallucinations (text added that was not in the transcript). In both cases, the DCMD confirmed the hallucinations were clinically accurate: Asciminib was referred to as “a novel ABL myristoyl pocket inhibitor”; “Paronychia” was included in the toxicity profile of amivantamab + lazerinib; to prevent future hallucinations, we refined our prompt to include “Only summarize the text that is given, don’t make any inferences”. Conclusions: Our use of GPT4DFCI significantly reduced the turnaround time for generating pathway meeting summaries. However, the process still required clinical expertise and review. Even in scenarios where there were no significant errors, human oversight improved the quality and consistency of the AI-generated content. In addition, using well-structured, specific prompts yielded more accurate responses. Our pilot shows that AI can enhance processes by efficiently summarizing information, but it should be used as a supplementary tool with human oversight.