Effect of Emphasizing Methodological Limitations of Primary Studies in a Systematic Review Summary: Findings from Two Randomized Controlled Trials

Background: Various organizations expend resources to communicate the findings of systematic reviews to non-specialist audiences, such as policymakers and the general public. The Campbell Collaboration does so with its plain language summaries, which aim to communicate complex methodological information in an accessible manner. However, the effectiveness of such summaries in communicating the overall quality of the primary studies included in a systematic review remains understudied.Objectives: This study aims to assess how the design of a systematic review summary affects readers' perception of methodological quality of primary studies, their perception of the evidence's definitiveness and their attitudes towards the policy intervention discussed in the review.Data and Methods: Using a web-based experimental design, two studies were conducted to examine how various presentation formats, including the emphasis on methodological limitations and the use of images, influence readers’ perceptions of information presented in a systematic review summary. Results: Emphasizing methodological limitations significantly reduces readers' perception of the quality of the methods used in the studies included in a systematic review summary. Yet, highlighting methodological limitations does not influence people's perception of the definitiveness of the review findings or their attitude toward related policies. Furthermore, removing the images from the summary’s standard design appears to have minimal effect on how readers interpret information about methodological limitations.Conclusions: Modifications to the traditional summary design can amplify the awareness of laypeople and civil servants regarding methodological limitations in the scientific evidence. However, their way of processing information about these methodological limitations does not automatically propagate the change in attitude regarding methods to a change in attitudes regarding study results and policy options.