AI-assisted evidence screening method for systematic reviews in environmental research: integrating ChatGPT with domain knowledge

Systematic reviews (SRs) in environmental science is challenging due to diverse methodologies, terminologies, and study designs across disciplines. A major limitation is that inconsistent application of eligibility criteria in evidence-screening affects the reproducibility and transparency of SRs. To explore the potential role of Artificial Intelligence (AI) in applying eligibility criteria, we developed and evaluated an AI-assisted evidence-screening framework using a case study SR on the relationship between stream fecal coliform concentrations and land use and land cover (LULC). The SR incorporates publications from hydrology, ecology, public health, landscape, and urban planning, reflecting the interdisciplinary nature of environmental research. We fine-tuned ChatGPT-3.5 Turbo model with expert-reviewed training data for title, abstract, and full-text screening of 120 articles. The AI model demonstrated substantial agreement at title/abstract review and moderate agreement at full-text review with expert reviewers and maintained internal consistency, suggesting its potential for structured screening assistance. The findings provide a structured framework for applying eligibility criteria consistently, improving evidence screening efficiency, reducing labor and costs, and informing large language models (LLMs) integration in environmental SRs. Combining AI with domain knowledge provides an exploratory step to evaluate feasibility of AI-assisted evidence screening, especially for diverse, large volume, and interdisciplinary studies. Additionally, AI-assisted screening has the potential to provide a structured approach for managing disagreement among researchers with diverse domain knowledge, though further validation is needed.