Implementation and Evaluation of an Open-Source Chatbot for Patient Information Leaflets

Accessing and understanding medication information can be challenging for many people, especially when patient information leaflets (PILs) are long, complex, and printed in small font. This study presents MediChat, an open-source, locally executable chatbot designed to provide reliable, easy-to-read answers to medication-related questions based exclusively on official PILs. MediChat follows a retrieval-augmented generation (RAG) architecture: PILs from the Austrian Medicinal Product Index are received via API, converted to text, split into overlapping chunks, embedded, and stored in a Chroma vector database. From there the top-k relevant chunks are retrieved, and Llama 3.1 generates German responses based on this evidence. The system was evaluated using a hybrid framework. Quantitatively, 200 yes/no questions across ten drugs were answered with 80% accuracy, overall precision 0.977, recall 0.686, F1-score 0.806, and a mean response time of 727 ms. Qualitatively, two personas were used in eight simulated dialogues. Response times were around 1.1–1.3 s, and task completion exceeded 85% with high ratings for relevance and quantity. These results indicate that an open-source RAG chatbot can deliver leaflet-grounded, user-friendly medication information and provide a reproducible template for future healthcare chatbot evaluations.