Automating postmarketing pharmacovigilance with large language model agents
Postmarketing pharmacovigilance relies on spontaneous adverse event reporting and labor-intensive observational studies, which suffer from underreporting and delayed risk detection. Large language model (LLM) agents could automate real-world evidence generation, but generalist agents make ad hoc analytical decisions that compromise epidemiological validity and reproducibility. Here we present MedCheckpoint, a multi-agent LLM system for automated postmarketing pharmacovigilance. MedCheckpoint grounds each analytical stage in a curated, source-traceable medical knowledge base, locks core study-design parameters before execution, performs patient-level analyses with human-written deterministic statistical tools, and enforces checkpoint rules that abort infeasible analyses, with optional human-in-the-loop review. We evaluated MedCheckpoint using a 54-question benchmark on a dual-purpose simulated electronic health record (EHR) testbed of 2.7 million individuals, designed with reliable clinical signals to evaluate routine analytical accuracy alongside spurious signals to stress-test system rigor. Across the benchmark, MedCheckpoint outperformed two coding-agent baselines in report quality across five dimensions rated by three independent LLM judges (overall means 4.81-4.88 versus 4.25-4.55 and 4.02-4.29) and yielded higher run-to-run consistency (mean within-question standard deviation 0.116-0.186 versus up to 0.440). Crucially, while baselines failed on up to 2 of 21 runs when handling infeasible comparisons or unmeasurable outcomes, MedCheckpoint produced no misleading failures across all seven stress-test trap questions with known data limitations. MedCheckpoint enables reliable, auditable, and scalable pharmacovigilance analysis.
Authors
- Celine Sze Ling Chui (ORCID: https://orcid.org/0000-0003-1513-8726)
- Ruibang Luo (ORCID: https://orcid.org/0000-0001-9711-6533)
- Yekai Zhou (ORCID: https://orcid.org/0000-0002-9260-0059)
- Zhenqin Wu (ORCID: https://orcid.org/0000-0002-0148-7055)
- Doris Yu
- Ruiheng Zhang
Institutions
- University of Hong Kong (HK)
Publication Details
- Journal
- medRxiv
- Published
- 2026-11-12
- DOI
- https://doi.org/10.64898/2026.11.10.26367711
- Primary Topic
- Pharmacovigilance and Adverse Drug Reactions
- Type
- preprint