Wrong field, wrong baseline, wrong performance rankings: Why publication-level normalization matters in research assessment exercises

In selected-output research assessment, a fixed submission quota draws on eligible publication portfolios that differ in size across fields. These unequal selection opportunities motivate normalizing selected-output scores and make the choice of reference class consequential. Italy’s 2020–2024 Valutazione della Qualità della Ricerca (VQR) explicitly normalizes selected-output scores by the researcher’s administrative area. This paper examines whether, within that framework, STEMM outputs should instead be normalized by the publication’s scientific field. We simulate a VQR-style assessment of Italian universities using Web of Science articles and reviews produced during 2020–2024 by 37,366 professors at 96 universities. We select an average of 2.5 outputs per professor under VQR-like constraints, assign VQR-like scores from citation percentiles, and compare university-area rankings under author-area and publication-field normalization. Scores are highly correlated, but rankings are not equivalent: Physics and Industrial and Information Engineering display substantial rank movements and quartile changes. Conditional on the VQR’s decision to normalize scores for cross-field institutional aggregation, a publication-field baseline is conceptually more coherent because it follows the object evaluated. It does not, however, by itself remove portfolio-size and selection effects.

Authors

Institutions

Publication Details

Journal
Journal of Informetrics
Published
2026-11-17
DOI
https://doi.org/10.1016/j.joi.2026.101892
Primary Topic
scientometrics and bibliometrics research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Wrong field, wrong baseline, wrong performance rankings: Why publication-level normalization matters in research assessment exercises

Ciriaco Andrea D’Angelo, Giovanni Abramo
Journal of Informetrics
scientometrics and bibliometrics research
article

Wrong field, wrong baseline, wrong performance rankings: Why publication-level normalization matters in research assessment exercises

Ciriaco Andrea D’Angelo, Giovanni Abramo
article en

Abstract

In selected-output research assessment, a fixed submission quota draws on eligible publication portfolios that differ in size across fields. These unequal selection opportunities motivate normalizing selected-output scores and make the choice of reference class consequential. Italy’s 2020–2024 Valutazione della Qualità della Ricerca (VQR) explicitly normalizes selected-output scores by the researcher’s administrative area. This paper examines whether, within that framework, STEMM outputs should instead be normalized by the publication’s scientific field. We simulate a VQR-style assessment of Italian universities using Web of Science articles and reviews produced during 2020–2024 by 37,366 professors at 96 universities. We select an average of 2.5 outputs per professor under VQR-like constraints, assign VQR-like scores from citation percentiles, and compare university-area rankings under author-area and publication-field normalization. Scores are highly correlated, but rankings are not equivalent: Physics and Industrial and Information Engineering display substantial rank movements and quartile changes. Conditional on the VQR’s decision to normalize scores for cross-field institutional aggregation, a publication-field baseline is conceptually more coherent because it follows the object evaluated. It does not, however, by itself remove portfolio-size and selection effects.

Journal of InformetricsVol. 20(4)
University of Rome Tor Vergata (IT), Mercatorum University (IT)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon