Automating postmarketing pharmacovigilance with large language model agents

Postmarketing pharmacovigilance relies on spontaneous adverse event reporting and labor-intensive observational studies, which suffer from underreporting and delayed risk detection. Large language model (LLM) agents could automate real-world evidence generation, but generalist agents make ad hoc analytical decisions that compromise epidemiological validity and reproducibility. Here we present MedCheckpoint, a multi-agent LLM system for automated postmarketing pharmacovigilance. MedCheckpoint grounds each analytical stage in a curated, source-traceable medical knowledge base, locks core study-design parameters before execution, performs patient-level analyses with human-written deterministic statistical tools, and enforces checkpoint rules that abort infeasible analyses, with optional human-in-the-loop review. We evaluated MedCheckpoint using a 54-question benchmark on a dual-purpose simulated electronic health record (EHR) testbed of 2.7 million individuals, designed with reliable clinical signals to evaluate routine analytical accuracy alongside spurious signals to stress-test system rigor. Across the benchmark, MedCheckpoint outperformed two coding-agent baselines in report quality across five dimensions rated by three independent LLM judges (overall means 4.81-4.88 versus 4.25-4.55 and 4.02-4.29) and yielded higher run-to-run consistency (mean within-question standard deviation 0.116-0.186 versus up to 0.440). Crucially, while baselines failed on up to 2 of 21 runs when handling infeasible comparisons or unmeasurable outcomes, MedCheckpoint produced no misleading failures across all seven stress-test trap questions with known data limitations. MedCheckpoint enables reliable, auditable, and scalable pharmacovigilance analysis.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-12
DOI
https://doi.org/10.64898/2026.11.10.26367711
Primary Topic
Pharmacovigilance and Adverse Drug Reactions
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Automating postmarketing pharmacovigilance with large language model agents

Celine Sze Ling Chui, Ruibang Luo, Yekai Zhou, Zhenqin Wu et al.
medRxiv
Pharmacovigilance and Adverse Drug Reactions
preprint

Automating postmarketing pharmacovigilance with large language model agents

Celine Sze Ling Chui, Ruibang Luo, Yekai Zhou, Zhenqin Wu, Doris Yu, Ruiheng Zhang
preprint en

Abstract

Postmarketing pharmacovigilance relies on spontaneous adverse event reporting and labor-intensive observational studies, which suffer from underreporting and delayed risk detection. Large language model (LLM) agents could automate real-world evidence generation, but generalist agents make ad hoc analytical decisions that compromise epidemiological validity and reproducibility. Here we present MedCheckpoint, a multi-agent LLM system for automated postmarketing pharmacovigilance. MedCheckpoint grounds each analytical stage in a curated, source-traceable medical knowledge base, locks core study-design parameters before execution, performs patient-level analyses with human-written deterministic statistical tools, and enforces checkpoint rules that abort infeasible analyses, with optional human-in-the-loop review. We evaluated MedCheckpoint using a 54-question benchmark on a dual-purpose simulated electronic health record (EHR) testbed of 2.7 million individuals, designed with reliable clinical signals to evaluate routine analytical accuracy alongside spurious signals to stress-test system rigor. Across the benchmark, MedCheckpoint outperformed two coding-agent baselines in report quality across five dimensions rated by three independent LLM judges (overall means 4.81-4.88 versus 4.25-4.55 and 4.02-4.29) and yielded higher run-to-run consistency (mean within-question standard deviation 0.116-0.186 versus up to 0.440). Crucially, while baselines failed on up to 2 of 21 runs when handling infeasible comparisons or unmeasurable outcomes, MedCheckpoint produced no misleading failures across all seven stress-test trap questions with known data limitations. MedCheckpoint enables reliable, auditable, and scalable pharmacovigilance analysis.

medRxiv
University of Hong Kong (HK)
Good health and well-being
Pharmacovigilance and Adverse Drug Reactions
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon