Evaluating\nGPT Models for Automated Literature Screening\nin Wastewater-Based Epidemiology

Methods to quantitatively synthesize findings across\nmultiple studies\nis an emerging need in wastewater-based epidemiology (WBE), where\ndisease tracking through wastewater analysis is performed at broad\ngeographical locations using various techniques to facilitate public\nhealth responses. Meta-analysis provides a rigorous statistical procedure\nfor research synthesis, yet the manual process of screening large\nvolumes of literature remains a hurdle for its application in timely\nevidence-based public health responses. Here, we evaluated the performance\nof GPT-3, GPT-3.5, and GPT-4 models in automated screening of publications\nfor meta-analysis in the WBE literature. We show that the chat completion\nmodel in GPT-4 accurately differentiates papers that contain original\ndata from those that did not with texts of the Abstract as the input\nat a Precision of 0.96 and Recall of 1.00, exceeding current quality\nstandards for manual screening (Recall = 0.95) while costing less\nthan $0.01 per paper. GPT models performed less accurately in detecting\nstudies reporting relevant sampling location, highlighting the value\nof maintaining human intervention in AI-assisted literature screening.\nImportantly, we show that certain formulation and model choices generated\nnonsensical answers to the screening tasks, while others did not,\nurging the attention to robustness when employing AI-assisted literature\nscreening. This study provided novel performance evaluation data on\nGPT models for document screening as a step in meta-analysis, suggesting\nAI-assisted literature screening a useful complementary technique\nto speed up research synthesis in WBE.