LR-AI: Evaluating an End-to-End LLM Pipeline from Search to Reading List for Systematic Reviews

Searching and screening procedures in a systematic review can take hundreds of person-hours. Most evaluations of large language models (LLMs) for this work test screening on a fixed set of papers, not the whole path from a research question to a reading list. We present LR-AI, an open pipeline that starts from a review's topic and, if any, its questions and eligibility criteria, and writes its own search queries. It searches scholarly databases, screens papers with an LLM, follows references and citing papers, and returns a ranked reading list of at most 300 papers. We evaluate it against five published reviews in cybersecurity, software engineering, medicine, psychiatry and neuroscience, using the 475 studies their authors included as the answer key. LR-AI finds 92% of these studies and places 66% in its lists, against 39% for a weak baseline that ranks the database results without LLM screening or citation searching, in lists capped at the same length. The screener keeps 93% of the included studies it sees, and most other losses come from two adjustable settings, a keyword filter on papers found through citations and the list cap. Across nine model settings screening the same papers, more capable models tended to keep a higher share of the selected reference papers in their lists, and the weakest kept the fewest. Repeated screens agree closely (mean pairwise κ of 0.79–0.87), and prior-knowledge checks show no consistent additional advantage from recalling the reviews.