Evaluating the accuracy and speed of eight deduplication tools: A comparative study

A key task in conducting systematic reviews is deduplicating the results from database searching. Deduplication using reference management software can be time-consuming and prone to error, while automated tools can be expensive and lack transparency. To support review teams, we evaluated eight deduplication tools: (1) The Automated Systematic Search Deduplicator (ASySD); (2) Covidence; (3) Deduklick; (4) EPPI-Reviewer; (5) PICO Portal; (6) Rayyan; (7) The Systematic Review Accelerator (SRA) Deduplicator: Focused; (8) The SRA Deduplicator: Relaxed. Five randomly selected Cochrane reviews had their searches rerun to create five gold standard sets. We compared the gold standard sets to the outputs of the eight deduplication tools and evaluated the results for: (1) unique records removed; (2) duplicate records retained; (3) time taken to deduplicate. Summed across all five reviews, the unique records removed in error ranged from 2 to 22. The three best tools were: (1) Rayyan; (2) Covidence; (3) SRA Deduplicator: Focused. The duplicate records retained in error ranged from 34 to 280, the three best tools were: (1) ASySD; (2) Rayyan; (3) EPPI-Reviewer. The time taken to deduplicate ranged from one minute to 20 hours and 34 minutes, the three fastest tools were: (1) SRA Deduplicator: Relaxed; (2) Deduklick; (3) Covidence. No tool performed so poorly that we don't recommend using it. But, as all the tools had strengths and weaknesses, some are expensive while others require large amounts of manual checking time, we recommend review teams compare the tools across all three outcomes and choose the tool that best suits their needs.