Is groundbreaking research disruptive? The covariation of disruption indices with researchers' self-assessments

Abstract Park et al. (2023) reported a decline in disruptive science since World War II, sparking debate about a perceived lack of major scientific achievements. Their study relied on the disruption index, a bibliometric indicator intended to identify disruptive patents and publications. However, this measure may miss substantial efforts contributing to scientific innovation, raising questions about its suitability for identifying major achievements. In this study, we tested whether the two most commonly used variants of the disruption index - the DI1 and the DI5 - correspond with researchers' self-assessments. Our findings show that researchers view groundbreaking science as combining novelty and importance, but disruption scores do not correspond to authors' self-assessments of their papers' groundbreaking character. This discrepancy may result from measurement weaknesses in the indices or from limitations in using author self-assessment as a benchmark. We conclude that further research is needed on disruption indices before they can be reliably used for research evaluation.