An investigation of discrepancies in outcome reporting and selective reporting bias in interrupted time series studies of health interventions: a methodological study

BACKGROUND: Selective reporting bias occurs when authors report outcomes or results based on the P value, magnitude or direction of the results. Selective reporting has not been examined in interrupted time series (ITS) studies. Therefore, we investigated (1) discrepancies between outcomes specified in protocols and those reported in the corresponding results reports, (2) completeness of results reporting, and (3) whether there was evidence of selective reporting bias in ITS studies. METHODS: We systematically searched for published peer-reviewed protocols of ITS studies of health interventions in 22 databases, and corresponding results reports addressing the protocol's primary ITS research question. We identified outcomes that were reported in the protocol and the corresponding results report(s), and all results reported for those outcomes. We defined a discrepancy as any outcome that was missing from the protocol or results report, or had its primacy reclassified (e.g., from primary to secondary) in the results reports compared to the protocol. Each result was classified as fully reported if it was reported with both an effect estimate and a measure of precision (e.g. confidence interval). Each result was also classified as favourable or not favourable to the interruption, based on its statistical significance and direction of effect. RESULTS: Our search for ITS protocols returned 4,590 abstracts. After excluding ongoing studies, protocols without published results for the primary research question, and records excluded for other reasons, we identified 44 ITS protocols (published 2010-2022) with 46 corresponding results reports. From these, we extracted 553 outcomes. Non-reporting of outcomes was prevalent, with 24% (132/553) of outcomes defined in the protocol not reported in the results report, and 60% (25/42) of protocols having at least one outcome not reported in the result reports. Among the 388 outcomes that could be assessed for discrepancies, 52% (202/388) had a discrepancy, affecting 74% (31/42) of studies. Only 28% (56/202) of discrepancies were justified by authors in the results reports. The association between a result favouring the interruption (based on statistical significance and direction of effect) and the result being fully reported was uncertain (OR = 1.06 [95% CI 0.74 to 1.53]). CONCLUSION: Non-reporting of outcomes and discrepancies in outcome reporting were prevalent. Pre-specifying outcomes in protocols and registries helps mitigate selective reporting. However, outcomes should be described in sufficient detail for readers to detect any changes.