Enhanced LLM-supported instructions for medication use through retrieval-augmented generation

Prescription systems support medication treatment by standardizing content, enhancing legibility and promoting correct dispensation. Challenges remain to personalize use instructions with sufficient clarity to meet patients' needs. Large Language Models (LLMs) are promising to improve communication in healthcare. This study investigates LLM Retrieval-Augmented Generation (RAG) drawing on patient information leaflets (PIL). We used a database with 119 outpatient scenarios to prepare prescriptions as input to LLMs. We evaluated patient instructions generated by open-source Llama3, using a standard prompt - model1, enhanced prompts with structured guidance - model2, and RAG-enhanced prompts incorporating PIL information from the Brazilian Health Regulatory Agency - model3. Five physicians independently scored the model-generated instructions, assessing adequacy, clarity, personalization and quality compared to reference texts; they reported their confidence level to perform the evaluation. Cosine similarity was computed between model-generated and reference texts. RAG introduction significantly improved instructions adequacy: median (interquartile range) 93.0 (40.0) vs. 94 (53.3) vs 100 (15.0), p = 0.001, and clarity 90.0 (60.0) vs. 92 (55.0) vs 95.0 (24.0), p = 0.012, for model1, model2, and model3 respectively. Enhanced prompt plus RAG in PIL reduced critical errors such as incorrect and incomplete instructions and factual inaccuracies. Prompt engineering positively impacted output, precluding the generation of vague and unsolicited instructions. Little impact of hallucination was found. Our findings underscore the role of AI in promoting safer medication use by incorporating authoritative medication information while maintaining concise and patient-friendly language. Human validation of output ensures error-free and safe implementation.