Comparative Evaluation of Commercial and Open-Source Large Language Models in a RAG-Based Chatbot for Antimicrobial Resistance Literature Analysis (Preprint)

<sec> <title>BACKGROUND</title> Antimicrobial resistance (AMR) poses a critical global health threat, undermining the efficacy of antibiotics and complicating clinical decision-making. Although scientific literature on AMR is extensive, retrieving and synthesizing relevant evidence remains time-consuming for clinicians and researchers. Recent advances in large language models (LLMs) offer opportunities to enhance access to domain-specific knowledge. However, the diversity of available models, ranging from open-source to commercial, necessitates a systematic comparison of their performance, cost, and scalability in real-world biomedical applications. </sec> <sec> <title>OBJECTIVE</title> This study describes the development of a Retrieval-Augmented Generation (RAG) chatbot for AMR literature analysis and compares multiple commercial and open-source LLMs in terms of accuracy, faithfulness, response time, and cost-efficiency. </sec> <sec> <title>METHODS</title> A corpus of 164 peer-reviewed AMR-related articles was compiled from Google Scholar and embedded into a ChromaDB vector database using OpenAI’s text-embedding-ada-002. The RAG chatbot was configured to run with five LLM backbones: GPT-4, GPT-4o, GPT-4o-mini, Claude 3.7 Sonnet, and LLaMA 4 Maverick. For each model, a temperature ablation study was performed to identify optimal performance. Evaluation metrics included correctness (pass rate and score), faithfulness, relevance, computational cost, and latency, using a synthetic ground truth dataset generated with GPT-4. </sec> <sec> <title>RESULTS</title> All models generated scientifically justified responses when integrated into the RAG framework. GPT-4 achieved the highest correctness score (94.7%) but incurred the highest cost, while GPT-4o delivered nearly identical accuracy at a ninefold lower cost and the fastest response time (3.88 s). LLaMA 4 Maverick and GPT-4o-mini offered lower accuracy but substantially reduced operational costs. Claude 3.7 Sonnet showed competitive accuracy but the least favorable cost-performance ratio. Qualitative analysis revealed differences in response style, detail, and structure among models. </sec> <sec> <title>CONCLUSIONS</title> A RAG-based chatbot can effectively support AMR research by delivering accurate, context-grounded, and scalable access to scientific literature. The comparative evaluation highlights trade-offs between performance, cost, and speed, guiding the selection of LLM architectures for clinical and research settings. Future work will focus on integrating language-specific embeddings and specialized domain agents to further enhance accuracy, adaptability, and clinical utility. </sec>