Expert-referenced evaluation of large language models for primary suspect drug identification in the FAERS database

Background Accurate identification of primary suspect drugs in adverse event reports is a critical component of post-marketing pharmacovigilance and serves as a foundation for downstream signal detection activities. However, causal attribution in spontaneous reporting systems such as the U.S. Food and Drug Administration’s Adverse Event Reporting System (FAERS) is inherently challenging. Traditional expert-based assessments are resource-intensive and difficult to scale, highlighting the need for complementary, reproducible approaches. This study aimed to evaluate the performance of large language models (LLMs) in identifying primary suspect drugs in adverse event reports against an expert reference standard. Methods Fifty FAERS case reports were randomly selected for evaluation. Three clinical pharmacy experts established the reference standard by adjudicating the primary suspect drug(s) for each case. LLM-based evaluations were conducted using multiple prompting strategies and model configurations. The performance of LLMs, pharmacy student evaluators, and original FAERS primary suspect designations (ROLE_COD) was assessed by calculating hit rates against the expert reference standard. Results The original FAERS designations demonstrated limited accuracy against the expert reference standard, with a hit rate of 0.778. In contrast, LLM-based assessments achieved consistently high hit rates ranging from 0.867 to 0.920 across diverse prompting strategies and model configurations. Student evaluators achieved a comparable hit rate of 0.872; however, LLM-based evaluations exhibited greater internal consistency and markedly shorter response times. Conclusion LLM-based evaluations demonstrated high accuracy against expert assessments and favorable efficiency relative to student evaluators. These findings highlight the potential utility of LLMs as scalable, readily accessible decision-support tools in pharmacovigilance workflows.