Benchmarking large language models for adverse drug reaction extraction in social media and clinical texts

Adverse Drug Reactions represent a significant threat to patient safety and remain a critical focus of pharmacovigilance, yet traditional extraction methods struggle with ambiguity, under-reporting, and noisy biomedical or social media texts. Large Language Models (LLMs) have emerged as a promising alternative due to their contextual understanding and biomedical pre-training, but their reliability in distinguishing true ADRs from spurious associations is still uncertain. This study aims to systematically evaluate the effectiveness of state-of-the-art open- and closed-source LLMs, including both general-purpose and domain-specific models, for ADR extraction. We benchmark GPT-3.5-turbo, GPT-4o-mini, Microsoft Phi-3-mini, LLaMA-3.2-1B, DeepSeek, and BioMistral-7B using two complementary approaches: fine-tuning and in-context learning (zero-shot, one-shot, and few-shot). Experiments are conducted on two benchmark datasets, CADEC and SMM4H, with evaluation based on strict and relaxed precision, recall, and F1-score metrics. Results show that GPT-4o-mini achieves the highest fine-tuning performance, with relaxed F1-scores of 79.06% on CADEC and 66.80% on SMM4H. In the five-shot in-context learning setting, it achieves 65.57% on CADEC and 49.34% on SMM4H, highlighting both progress and ongoing limitations. These findings demonstrate that while LLMs hold significant promise for enhancing ADR extraction, they still face challenges in capturing the complexity and variability of real-world ADR expressions, indicating that further model adaptations and domain-specific refinements are required to improve their effectiveness in pharmacovigilance applications. • The study evaluates state-of-the-art open- and closed-source LLMs for ADR extraction. • Fine-tuning and in-context learning are compared on CADEC and SMM4H datasets. • GPT-4o-mini fine-tuning achieves the best performance with the highest relaxed F1-scores. • Results reveal progress but highlight challenges in handling noisy real-world texts. • Findings emphasize the need for domain-specific adaptations to enhance pharmacovigilance.