S848 Evaluating a Multi-Agent LLM Pipeline With Human-in-the-Loop for Drafting Patient Education Handouts From Eosinophilic Esophagitis Guidelines

Introduction: Printed patient handouts are essential in clinical care but are often delayed following updates to society guidelines. Large language models (LLMs) may help accelerate this process. This study evaluates a multi-step agentic workflow, powered by Gemini 2.0 and guided by a human expert, to extract key information from the 2025 American College of Gastroenterology guidelines for eosinophilic esophagitis (EoE) and generate patient-ready handouts. Methods: The guidelines were processed using a multi-step agentic pipeline with 4 artificial intelligence (AI) agents handling extraction, simplification, and formatting. A board-certified gastroenterologist provided oversight at key points. The simplifying agent optimized content for patient readability based on the Simple Measure of Gobbledygook (SMOG) index. Python 3.7 automated the process. Eight handout drafts were generated without manual editing, then evaluated by 7 healthcare professionals—an esophagologist (45 years’ experience), 2 attendings, 1 intern, 2 APPs, and 1 motility nurse. Each completed a 19-item Microsoft form, rating 6 domains (1-5 Likert scale), plus overall satisfaction and comments. Results: A total of 53 evaluations were submitted. Average Likert scores across 6 categories ranged from 4.4 to 4.9; individual handout averages ranged from 4.5 to 4.8. Overall satisfaction was high: 60% very satisfied, 32% satisfied, and 8% neutral. For clinical readiness: 40% selected “ready now,” while 45% selected needing minor, moderate (13%), or major changes (2%). Average SMOG score was 12.4, exceeding the <7 target. However, further simplification consistently led to loss of essential clinical content. Reviewers praised layout, tone, and clarity, while suggesting minor improvements in medical precision and depth. Conclusion: Prior LLM studies on handouts exist, but our approach adds unique strengths. Specialized agents reduced hallucinations and improved task fidelity. Restricting inputs to society guidelines minimized outdated content. A human-in-the-loop framework preserved oversight while AI handled the time-intensive drafting process. The model struggled to balance readability and information retention, but reviewers still found the materials clear and appropriate. Future work includes testing patient comprehension, feedback from non-healthcare providers, and adapting the workflow to other diseases. This tool shows strong promise for rapidly drafting high-quality patient materials requiring minimal modification across clinical contexts.