Retrieval-Augmented Generation for Pharmacopoeia: Application and Evaluation

Pharmacopoeia documents are used in pharmaceutical companies as references for ensure safe handling of chemical compounds. However, manual review of these documents can be time-consuming and prone to human error. To assist users in working with these documents, we designed PharmChat, a retrieval-augmented generation (RAG)-based system which enables efficient question-and-answer interactions with multiple short pharmacopoeia documents using natural language. In contrast to other conventional RAG systems, PharmChat converts original pharmacopoeia into separate documents to avoid context mixing during document chunking. PharmChat leverages embedding-based similarity searches between the documents as knowledge-base and user queries using an open-source vector database, ChromaDB. Upon receiving a user query, the most relevant chunks are retrieved and provided as context for a large language model (LLM) to be used as reference to minimize hallucination. Experimental results with real pharmacopoeia short documents demonstrate that our PharmChat system provides quick response times and achieves a BERT-F1 score of 76%.