The Measurement of Data Science Outcomes a Matter of Accuracy

Purpose: This study reviews the literature on the outcomes obtained from the use of data science in industry and academy. The search yielded 36,772 articles that comprised the total base of articles published in events and journals from 2013 to 2021. Methods: This systematic literature review involved a mapping study. The protocol covers the general theme, research questions, data sources, search strings and inclusion and exclusion criteria for primary sources. Findings: Our research question (“what are the means adopted to evaluate the results of data science implementations?”) revealed that the measurement of data science outcomes has been conceived as a specific algorithmic performance, that is, a measure of accuracy, but accuracy has a direct relationship with data and models, and an indirect relationship with reality, which suggests an overvaluation of accuracy, and data science accuracy-based outcomes, to the detriment of a concern with (a) the representation of the reality of interest, and (b) measurement as an important component of a distinct science. Implications: The absence of broad/generic measurement instruments makes data science results non-comparable between instances of implementation, which ultimately limits the development of such emergent science.