A Fine-Tuned Multi-Agent Large Language Model Platform with Retrieval-Augmented Generation for Clinical Case Report Analysis

This study presents a prototype for the structured analysis of PubMed case-report titles, keywords and abstracts, combining domain fine-tuning, retrieval-augmented generation (RAG) and five specialist-role agents with a synthesis agent. DeepSeek-R1-Distill-Qwen-7B was fine-tuned using QLoRA on 8999 reports with teacher-generated targets. The retrieval index contains 7679 chunks from 13 medical sources. On 501 held-out reports, symmetric final-answer-only scoring, with missing final answers assigned zero, showed increases after fine-tuning in ROUGE-1 (+0.0253, 95% CI +0.0107 to +0.0394) and structural completeness (+0.0371, +0.0160 to +0.0579). The mean BERTScore difference was +0.0035 (−0.0085 to +0.0153), providing inconclusive evidence of a mean improvement. Among the tested platform configurations, retrieval showed no clear benefit on the reported final-answer reference metrics, and the specialist panel scored below a single structured agent. Final-answer results for the full platform versus the model alone were mixed. The full platform used 11.7 times as many tokens and 5.9 times the latency of the fine-tuned model alone. A blinded LLM-as-a-judge audit of 1941 statements found that 20–24% were unsupported or contradicted by the supplied sources; among checkable factual statements, this proportion was 24.5–28.5%. Different report requirements and a shared 1024-token generation limit constrain component-level interpretation. The results concern the evaluated configurations, reference similarity, format and source agreement; they do not establish improved diagnostic accuracy or clinical safety.