Welcome to the Metrics Jungle: Organizational Stakeholder Perspectives on Evaluation of News Recommender Systems in Industry

News recommender systems (NRSs) are increasingly implemented in the news industry to enhance digital distribution, yet their development and evaluation are shaped by a complex interplay of organizational stakeholders with diverse objectives. This study investigates the real-world complexities of NRS implementation, considering the multitude of objectives on the NRS provider side by exploring how different stakeholder groups —from journalistic, commercial, and tech logic— conceptualize NRS goals and assess their performance. Through 22 semi-structured expert interviews across four news organizations in Europe, we uncover a ‘jungle of metrics’ where an overabundance of data insights does not necessarily lead to better decision-making. While traditional accuracy-based metrics like click-through rate (CTR) remain widely used, they fail to capture the broader strategic objectives that guide NRS adoption in industry. This paper identifies a critical gap between current academic evaluations of NRS and the business key performance indicators (KPIs) used in practice. Rather than evaluating NRS in isolation, stakeholders assess their contribution to overarching organizational goals, including reach, engagement, conversion, and retention. Our findings highlight the need for a more nuanced, multi-stakeholder approach to NRS evaluation that aligns with both business imperatives and journalistic values.