TIP-28. Fine-tuning large language models using NCCN guidelines to aid primary CNS cancer treatment decisions

Abstract Managing primary central nervous system (CNS) cancers presents a challenge, particularly for Rural or general oncologists (without access to a neuro-oncologist) who do not frequently encounter these conditions. NCCN guidelines continue to change compared to when oncologists were trained, which can complicate clinical decision-making. Remaining up-to-date with numerous clinical trials and evolving treatment is also a significant problem, especially when CNS cancers are rarer than other forms of cancer. This ongoing trial evaluates the effectiveness of a large language model (LLM), fine-tuned using NCCN guidelines specific to primary CNS cancers. Using de-identified patient information, the subjective and objective data will be utilized in a cross-sectional design, comparing Chat GPT 4 to an LLM trained on NCCN guidelines. Board-certified neuro-oncologists will assess these AI-generated recommendations using a Generative AI Performance Score (G-PS) calculator, which considers guideline-based therapy, hallucinations/inaccuracies, and clinical rationality, standardized to a score of -1 to 1. We are seeking to evaluate the model’s capacity to produce accurate, NCCN-concordant initial treatment recommendations compared to LLMs that are not tied to the NCCN guidelines. We will also assess the rationale of LLMs trained on NCCN guidelines, considering patient information, quality of life, and relevant clinical trial data. Enrollment and evaluations are currently ongoing. Hallucinations or inaccuracies are a significant factor in clinical applications, and by also considering the amount, we will further demonstrate that LLMs trained on properly vetted data are superior to those trained on generalized public data. Preliminary conclusions are not yet available, but the initial assessment looks promising. Results will be reported following the completion of data collection and analyses, potentially demonstrating improvements in clinical decision-making, treatment accuracy, and patient care for the management of primary central nervous system (CNS) cancer, and that LLM should be further developed.