MENTOR: An Evidence-Informed Large Language Model-Based Multi-Agent System for Efficient Selection of Institution-Specific Negative Control Outcomes

Objective Negative control outcomes (NCOs) help detect and correct systematic error in observational studies, yet their selection remains largely manual. The Common Evidence Model (CEM) can generate candidate lists, but researchers must still screen them for plausibility as true nulls and for sufficient observation in each institution's data. We developed and evaluated MENTOR (Multi-agent Evidence-informed Negative conTrol Outcome selection with Reasoning), a multi-agent system of large language models (LLMs) that combines CEM evidence with LLM reasoning to select institution-specific NCOs. Methods Four LLM agents operate on an institution's Observational Medical Outcomes Partnership common data model. A Curator summarizes drug labels, a Suggester assesses locally frequent candidates with CEM evidence and a first-pass LLM judgment, an Adjudicator resolves their disagreements, and a Diagnostician relaxes the patient-count threshold until a target number of NCOs is met. Using Qwen3.6-27B, we evaluated 349 candidates for three drug pairs in a Korean tertiary hospital's electronic health records (2.9 million patients) against a blinded LLM judge from a different model family, validated by three experts on 90 candidates. We compared accuracy, positive predictive value (PPV), and sensitivity with CEM-only and first-pass-LLM-only selection. Results MENTOR proposed 243 NCOs versus 121 by CEM-only, with higher accuracy (0.83 [0.79-0.87] vs. 0.53 [0.48-0.58]), sensitivity (0.89 vs. 0.41), and PPV (0.86 vs. 0.79). Judge-expert agreement was κ=0.63. Adjudication, needed for 44.1% of candidates, added little accuracy, and only with reasoning mode. Concordant acceptances, which bypass it, contained about half of the false positives. Only 7%-9% of CEM-proposed outcomes had at least 10,000 patients at the institution. Conclusion Combining LLM judgments with CEM evidence and local data yielded more NCOs, with higher PPV, than CEM alone. Restricting reasoning to discordant candidates kept computation time below 40% of that with reasoning in all agents. Every decision comes with its evidence and rationale for investigators' final review.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-12
DOI
https://doi.org/10.64898/2026.11.10.26367768
Primary Topic
Advanced Causal Inference Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

MENTOR: An Evidence-Informed Large Language Model-Based Multi-Agent System for Efficient Selection of Institution-Specific Negative Control Outcomes

Minkook Son, Junhyuk Chang, Jaehyeong Cho, Chungsoo Kim et al.
medRxiv
Advanced Causal Inference Techniques
preprint

MENTOR: An Evidence-Informed Large Language Model-Based Multi-Agent System for Efficient Selection of Institution-Specific Negative Control Outcomes

Minkook Son, Junhyuk Chang, Jaehyeong Cho, Chungsoo Kim, Kyungseon Choi, YeongHyeon Kim, Taeseok Yang
preprint en

Abstract

Objective Negative control outcomes (NCOs) help detect and correct systematic error in observational studies, yet their selection remains largely manual. The Common Evidence Model (CEM) can generate candidate lists, but researchers must still screen them for plausibility as true nulls and for sufficient observation in each institution's data. We developed and evaluated MENTOR (Multi-agent Evidence-informed Negative conTrol Outcome selection with Reasoning), a multi-agent system of large language models (LLMs) that combines CEM evidence with LLM reasoning to select institution-specific NCOs. Methods Four LLM agents operate on an institution's Observational Medical Outcomes Partnership common data model. A Curator summarizes drug labels, a Suggester assesses locally frequent candidates with CEM evidence and a first-pass LLM judgment, an Adjudicator resolves their disagreements, and a Diagnostician relaxes the patient-count threshold until a target number of NCOs is met. Using Qwen3.6-27B, we evaluated 349 candidates for three drug pairs in a Korean tertiary hospital's electronic health records (2.9 million patients) against a blinded LLM judge from a different model family, validated by three experts on 90 candidates. We compared accuracy, positive predictive value (PPV), and sensitivity with CEM-only and first-pass-LLM-only selection. Results MENTOR proposed 243 NCOs versus 121 by CEM-only, with higher accuracy (0.83 [0.79-0.87] vs. 0.53 [0.48-0.58]), sensitivity (0.89 vs. 0.41), and PPV (0.86 vs. 0.79). Judge-expert agreement was κ=0.63. Adjudication, needed for 44.1% of candidates, added little accuracy, and only with reasoning mode. Concordant acceptances, which bypass it, contained about half of the false positives. Only 7%-9% of CEM-proposed outcomes had at least 10,000 patients at the institution. Conclusion Combining LLM judgments with CEM evidence and local data yielded more NCOs, with higher PPV, than CEM alone. Restricting reasoning to discordant candidates kept computation time below 40% of that with reasoning in all agents. Every decision comes with its evidence and rationale for investigators' final review.

medRxiv
Seoul National University (KR), Yale University (US), Keimyung University Dongsan Medical Center (KR), Dong-A University (KR), Keimyung University (KR), Ajou University (KR)
Good health and well-being
Advanced Causal Inference Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon