Measuring the effect of structural differences between Web of Science and Scopus on research impact assessment

• Database choice impact on MNCS through structural differences is modeled. • Benchmark component serves as a proxy to measure (WoS, Scopus) database structural bias. • Novel CBDR metric developed and validated to quantify this benchmark effect. • CBDR-based analysis of Hungary’s output by fields, HEIs, and journal tiers. Our work elaborates on the study of how the choice of databases affects the measurement of scientific impact. We address a gap in the analysis of the “crown indicator”, the Normalized Citation Score (NCS), when applied in practical settings in different database contexts. It is shown that database-specific structural features have a profound effect on the differences in NCS values across databases, and that the benchmark component of the metric can be used as a proxy for quantifying the effect of such differences (“benchmark effect”). The primary aim of our analysis was the measurement of the effect of benchmark differences on impact scores at various levels of assessment. To that end, we developed and theoretically validated a novel metric for quantifying the effect of benchmark differences on NCS discrepancies across databases (Citation-Benchmark Differentials Ratio, CBDR). We applied the CBDR metric on the five-year country-level publication output of Hungary, retrieved from WoS and Scopus. Measurement results were subjected to statistical analyses to reveal the potential benchmark effect in various aggregations of the publication record, especially by research fields, research organizations (HEIs), journal quartiles, and publication years. Results show that a considerable fraction of the country-level output is affected by the benchmark effect, and no substantial differences emerge between the units of assessment in any aggregation under study. It is demonstrated that database-specific citation counts and journal coverage only partially account for the differences between WoS- and Scopus-based normalized citation scores.