Utilizing large language model technology to gain actionable insights into incoming patient phone call trends.

615 Background: Tennessee Oncology (TO) receives an average of 100,000 incoming phone calls to its Medical Oncology clinic locations each month. All incoming clinic calls are routed through a live operator workflow to route the calls to their appropriate destination. Only 31% of the calls get routed and classified into TO’s clinical triage system, leaving 69% of incoming calls without actionable data or monitored follow-up. Because every call is recorded and transcribed, we explored whether a large language model (LLM) could automatically classify these transcripts and surface useful operational insights. Methods: A secure, on-premises Llama-3 2-b parameter model hosted in the OneOncology/TO data warehouse was fine-tuned with custom prompts to: (1) extract caller identifiers and (2) assign each call a reason category and sub-category. Prompt iterations were evaluated through repeated manual audits; misclassifications informed prompt refinement until performance stabilized. 35,000 call transcripts, from January 17, 2025 to February 14, 2025, were processed through the final LLM setup. Blinded human audits of randomly selected outputs provided the reference standard for model accuracy. Results: Analyzing the call reasons, over 16,500 (47.4%) of the 35,000 transcripts were found to be related to a scheduling need. The scheduling category significantly outweighed the other identified categories, medication refill (12.4%), other (12.3%), lab/test results (7.1%), referral questions (5.1%), medical records (4.6%), orders questions (4.4%), symptom management (3.9%), and insurance/billing questions (2.8%), which drew attention to the scheduling volume. The scheduling subcategories were analyzed; rescheduling accounted for 9,400 (57.8%) of the 16,500 scheduling categorized calls. Internal scheduling needs (23.1%), external scheduling needs (8.4%), new patient scheduling (8.3%), and other scheduling (2.5%) made up less than half the scheduling subcategories. The model classified 98% of transcripts correctly. All audited misclassifications stemmed from ambiguous or inaudible audio rather than model hallucination. Conclusions: With careful prompt engineering and human-in-the-loop validation, an LLM can convert large volumes of unstructured call transcripts into reliable, actionable data. Discovering that more than half of all calls were related to scheduling, particularly rescheduling, prompted TO leadership to launch immediate process-improvement efforts and embed these findings in a 2025 ASCO Quality Training Program (QTP) project.