Optimising the clinical application of rheumatology guidelines using large language models: a retrieval-augmented generation framework integrating EULAR and ACR recommendations

<h2>ABSTRACT</h2><h3>Objectives</h3> Timely access to current rheumatology guidelines at the point of care is challenging. We aimed to develop and evaluate the first retrieval-augmented generation (RAG) system designed for adult rheumatology, integrating European Alliance of Associations for Rheumatology (EULAR) and American College of Rheumatology (ACR) guidelines to provide rheumatologists with timely evidence-based recommendations. <h3>Methods</h3> EULAR and ACR management guidelines were selected by rheumatologists based on their clinical relevance for decision-making and processed. A RAG system was implemented. To evaluate it, 10 questions per guideline were generated using ChatGPT 4.5. Answers to these were produced by ChatGPT-o3-mini with context retrieval (RAG) and without (baseline). Performance was assessed by an Large language model (LLM)-as-a-judge (Gemini 2.0 Flash) using a 5-point Likert scale across 5 dimensions: relevance, factual accuracy, safety, completeness, and conciseness; it also determined preference between the RAG and baseline responses. For validation, 2 rheumatologists independently evaluated a random sample of questions (15%) on the same domains. Statistical significance was established using the Wilcoxon signed-rank and binomial tests. <h3>Results</h3> Seventy-four guidelines were included, yielding 740 evaluation questions. The LLM-as-a-judge evaluation showed the RAG system significantly outperformed the baseline across all criteria (<i>P</i> < .001). Manual evaluation confirmed these findings (<i>P</i> < .001) for accuracy, safety, and completeness. The RAG system was significantly preferred by the LLM-as-a-judge in 92.8% of comparisons and by the human evaluators in 71.2% to 74.8% (<i>P</i> < .001). <h3>Conclusions</h3> Developing a RAG system integrating extensive EULAR/ACR rheumatology guidelines improves answer quality compared to a baseline LLM. This evaluation provides a robust foundation for reliable, artificial intelligence-driven clinical decision support tools designed to enhance evidence-based practice by providing clinicians with rapid, context-aware access to recommendations.