The performance of covidence: an artificial intelligence-based tool for title and abstract screening in a breast cancer evidence-based clinical practice guideline

Clinical practice guidelines (CPGs) support evidence-based care but are time-consuming to develop. This study aimed to compare artificial intelligence (AI)-assisted versus manual title and abstract screening (Stage I) in Covidence using data from a published breast-cancer CPG. This systematic review (SR) included 8,774 articles identified through a medical literature search, after removing duplicates. Three article subsets ( n =500, 1,000, and 2,000) were randomly selected from 8,774 articles to perform 30, 30, and 10 trials, respectively, independent Stage I, AI-assisted trials. The primary outcome of each trial is workload savings achieved through AI-assisted identification of 95% and 100% relevant articles (i.e., sensitivity), and 100% of finally-included articles. The secondary outcome is missed finally-included articles when the sensitivity of 95% was reached for each subset. At the sensitivity of 95%, 100% relevant articles and 100% finally-included articles were identified, median (minimum, maximum) workload savings are 40.7% (4.4%, 59.4%), 25.0% (0.4%, 55.2%), 57.6% (6.2%, 76.4%) for n =500; 38.3% (6.2%, 54.0%), 17.3% (0.0%, 39.1%), 63.9% (0.4%, 77.5%) for n =1,000; 16.6% (10.8%, 41.8%), 4.4% (0.3%, 20.9%), 17.9% (0.8%, 64.6%) for n =2,000 respectively. Covidence’s performance does not improve as the size of the subsets increases for a CPG with multiple complicated research questions. A potential positive correlation between the proportion of relevant articles in initial training of Covidence and workload savings at Stage I across all 70 trials. At 95% sensitivity, 5 trials missed 1 article ( n =500); 2 trials missed 2 articles and 1 trial missed 1 article ( n =1,000); 1 trial missed 3 articles, and 5trials missed 1 article ( n =2,000). AI-assistance in Covidence for Stage I screening shows both promise and pitfalls in the SR for a breast cancer CPG on a complex topic. Further prospective research is needed to better understand the performance of AI-assistance in Covidence and the intricacies of CPG topics.