Accounting for Retrospective Bias in Classification Systems of Cultural Products

Empirical analyses of cultural production often rely on categorizations assigned after the time when the products were released in the market. This late – retrospective – classification can generate measurement bias, and obscure relationships between products’ positioning and important outcomes such as their market performance or innovativeness. One such outcome with broad practical and theoretical relevance is audience appeal associated with the so-called category-spanning discount (e.g., Hsu, 2006). We closely replicate empirical results from Hsu’s (2006) key study of the genre-spanning discount on expert and consumer ratings of feature films. We then extend the study to an earlier period to find that the genre-spanning effect declines in magnitude and loses statistical significance when using retrospective categorization, providing suggestive evidence for measurement bias. Using natural language processing, we develop a methodological tool that harmonizes between classification systems generated at different points in time. We find that our harmonizing tool recovers the magnitude and statistical significance of the genre-spanning discount. Our methodology can more generally help to reduce bias in measuring relationships between category systems, their underlying concepts, and product outcomes.