Wealth of information and poverty of attention? Managing the inherent challenges of large scoping reviews

Data, as the writer Mark Twain phrased it, “is like garbage. You’d better know what you are going to do with it before you collect it.” This provoking line includes earnest wisdom: it is not the lack of data that is troublesome; rather, it is the vast amount of data creating noise that compromises our ability to separate the relevant from the irrelevant. Scoping reviews can be dangerous pieces of research. On the one hand, they are incredibly useful for systematically identifying and mapping the available evidenced on a particular topic.1 This utility is evidence by their increasing popularity, including in JBI Evidence Synthesis, which published 3 scoping reviews in 2017 versus 17 in 2023. On the other hand, the extensive amount of evidence found can be a double-edged sword. While the best areas of exploration for scoping reviews are research topics on which little is known, the amount of information available for building the dataset on a chosen topic may surprise even an experienced review enthusiast. With the aim of broadly mapping the available evidence, scoping reviews can snowball; for example, Tricco et al.1 examined 499 scoping reviews in which the mean number of included studies was 118, with one review containing 2600 studies! Clearly, these reviews require a lot of software power, brain power, and careful planning, in addition to “good nerves” and vigor, as experienced by yours truly when conducting a large scoping review with a dataset of 242 studies, published in this issue.2 Despite continuously evolving guidance for scoping reviews in general,3–7 until recently, there was no specific guidance for conducting large scoping reviews. Now this gap in the guidance has been addressed. In a commentary paper on conducting large scoping reviews, Alexander et al.8 define a scoping review as large if it includes approximately 100 sources of evidence or more, while also asserting that the volume of data to be extracted and the complexity of analyses can impact the breadth of the review. Deep content analysis examining meanings, themes, and patterns is always more complex than simple descriptive analytics. At the beginning of their work, reviewers presumably have a fairly good idea about the complexity of the analysis to be done, given that they should know what they are looking for from the review. What is unknown, however, is the amount of data available on the topic. Therefore, a wise way to approach a scoping review is to first create a publicly available protocol, with a pilot literature search and draft data extraction tool. Although this does involve plenty of work, it forces reviewers to define clear research questions and develop an appropriate search strategy. Maintaining a broad scope while keeping a clear focus is an important balance to strive for. An ideal review team consists of subject expert(s), review methodology expert(s), information specialist(s), and, potentially, the knowledge users. While a bigger review team can manage the search and data extraction quickly, a smaller team can work in a more agile manner, keeping the work well coordinated and effective. A small review team should not be overwhelmed by large scoping reviews and should not be deterred by perceived extensivity of the work. For our large scoping review, an a priori protocol9 was built by the team consisting of 2 subject experts in health economics, 1 review methodology expert, and 1 information specialist. The subject experts knew the phenomena of interest, and the review specialist helped to set the phenomena in the context of a scoping review. The role of the information specialist was important since careful development of the search strategy helped to narrow down the number of studies to be included. By publishing the protocol, knowledge users were also involved, as the peer review process pushed us to justify our approach and offered insights that improved the protocol. Although the first steps of conducting a large scoping review are similar to those of a “normal-sized” scoping review, the difference appears with the extensive dataset collected. Careful planning helps, but leave some room for exploration. In our review, data extraction from a large number of studies required systematically defining what information was to be collected, which was done in the a priori protocol. But to allow some exploration, a test data extraction from a smaller sample of studies was conducted, which clarified the optimal way of organizing and collecting the data available from the included studies, and revealed the needed modifications from the original data extraction form. Only after taking a sneak peek of the collected data were we able to finalize a feasible data extraction tool for the review. When the tool was finalized, the analysis of the data did not feel like such a huge task anymore. A concrete structure with sufficient flexibility surely supports reviewers navigating an abundance of data. The guidance from Alexander et al.8 is appreciated. As they suggest, review teams should think ahead about handling the data when planning a scoping review. More concrete guidance for building a data extraction tool would be useful, as this is the foundation for the data analysis, which generally is more complicated in large scoping reviews. With careful planning, conducting a large scoping review is achievable. It may be scary in the beginning and messy in the middle, but with clear guidance, large scoping reviews can be beautiful in the end.