Editors' Notes: Incorporating Open Science Into Evaluation

In an era where transparency, collaboration, and accessibility are increasingly valued across scientific disciplines, the principles of open science are becoming more relevant than ever. Open science, as defined by UNESCO, represents a broad and inclusive framework that aims to democratize scientific knowledge. It advocates for making scientific data, methodologies, and outputs accessible to all, thereby fostering a more collaborative and inclusive scientific community. This movement, rooted in the desire to enhance the credibility, replicability, and societal impact of research, has grown in response to significant challenges in the scientific community, including issues of reproducibility, data accessibility, and the prevalence of questionable research practices. The field of evaluation—with its unique blend of scientific inquiry and practical application—stands at a critical intersection where the adoption of open science practices could lead to substantial advancements. Evaluation, by its nature, involves rigorous analysis and decision-making processes that can benefit from the core tenets of open science: transparency, accessibility, credibility, and reproducibility. The parallels between evaluation and research, coupled with the increasing scrutiny on research practices across disciplines, suggest that it is in evaluators' best interest to be keenly aware of the open science movement and consider its implications for their work. This special issue explores the integration of open science practices into the field of evaluation. It aims to illuminate the potential benefits and challenges of applying open science principles to evaluation, while also acknowledging that open science is not a panacea. As with any methodological shift, the adoption of open science must be approached thoughtfully, recognizing the diversity of evaluation contexts and the varying applicability of open science practices. The articles in this issue dive into various aspects of how open science can be part of evaluation practice. From discussions on the alignment of open science principles with established evaluation standards, to explorations of specific open science practices such as data sharing, preregistration, and open access publishing, this issue provides an overview of the current landscape. It also highlights the potential for evaluators to contribute to the broader open science movement, particularly in areas related to research evaluation (i.e., evaluation of research processes and products) and the assessment of scientific impact. As you engage with the articles in this special issue, we invite you to consider how the principles of open science might influence your own evaluation work. The goal is not to prescribe a one-size-fits-all approach, but rather to encourage a reflective and adaptive application of open science practices that can enhance the rigor, transparency, and societal relevance of evaluation. The future of evaluation, much like the future of science, may be shaped by the principles of openness, collaboration, and inclusivity—principles that this issue seeks to bring to the forefront of our professional discourse. In these Editors' Notes, we first provide a foundational overview of core concepts, principles, and practices in open science so that readers have a common foundation for understanding the articles in this issue. The latter part of this article offers readers an overview of the issue and briefly discusses the future of open science in evaluation. Readers can access preprints, data, and other materials related to this NDE issue through the Open Science Framework at https://osf.io/p2dqs/. …an inclusive construct that combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone, to increase scientific collaborations and sharing of information for the benefits of science and society, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors beyond the traditional scientific community. It comprises all scientific disciplines and aspects of scholarly practices, including basic and applied sciences, natural and social sciences and the humanities, and it builds on the following key pillars: open scientific knowledge, open science infrastructures, science communication, open engagement of societal actors and open dialogue with other knowledge systems. (UNESCO 2021) UNESCO further articulates the core values that underpin open science, including quality and integrity, collective benefit, equity and fairness, and diversity and inclusiveness. These values are supported by guiding principles such as transparency, scrutiny, critique, and reproducibility; equality of opportunities; responsibility, respect, and accountability; collaboration, participation, and inclusion; flexibility; and sustainability. Open science is a broad and multifaceted concept, with various schools of thought contributing to its development (Fecher and Friesike 2014). While many practices associated with open science have existed for decades, the early 2000s and 2010s marked a significant shift toward the open science movement. This shift was catalyzed by events such as the publication of Bem's 2011 article, which controversially claimed to prove the existence of extrasensory perception despite employing highly questionable research practices like p-hacking. The lack of replication in various fields, including psychology, medicine, physics, and economics, further fueled concerns (Baker 2016). Additionally, research has shown that researchers often have significant flexibility—or “degrees of freedom”—in their work, which can increase the likelihood of false-positive results (Simmons et al. 2011). Many researchers have also admitted to engaging in questionable research practices, such as hypothesizing after results are known (HARKing), excluding data after viewing results, and collecting more data after initial results (John et al. 2012). Compounding these issues, much research suffers from insufficient power to detect the minimum effect size of interest (Bezeau and Graves 2001), and attempts to reproduce studies often fail due to the unavailability of data upon request (Tedersoo et al. 2021). In response to these challenges, a range of initiatives have emerged to promote open science. These initiatives include the EU's adoption of Open Science in 2016 (Miedema 2022), the San Francisco Declaration on Research Assessment (DORA; Butler-Adam 2017), the Pan-African Open Science Platform (AOSP; http://africanopenscience.org.za/), the Global Open Science Cloud (GOSC) launched by CODATA (https://codata.org/initiatives/making-data-work/global-open-science-cloud/), the Global South Committee for Open Science launched by eLife (https://elifesciences.org/community/global-south-committee), and cOAlition S (https://www.coalition-s.org/about/). For readers interested in more on the history or philosophy of open science, we suggest reading Miedema (2022) and Leonelli (2023). It is crucial to recognize that open science will not resolve all the challenges in the scientific enterprise. The diverse practices under the umbrella of open science mean that it is not a one-size-fits-all approach (Prosser et al. 2021) or an all-or-nothing proposition (Kathawalla et al. 2021; Syed 2019). Moreover, open science is not without its critiques and controversies. Concerns include whether open science stifles creativity (Vazire 2018), whether it is rooted in neoliberal (e.g., free-market, capitalistic, antilabor, deregulation) ideologies (Tunç et al. 2021), and issues surrounding the exclusionary behaviors of some open science advocates—often referred to as “bros”—who may be condescending, aggressive, and exclusionary in their discourse. These behaviors can disadvantage and inhibit the participation of women, early career researchers, and other marginalized groups in the open science movement (Pownall et al. 2022; Whitaker and Guest 2020). Given these complexities, evaluators should consider how open science principles, practices, and approaches can be thoughtfully incorporated into their practice. While open science is not a cure-all for the challenges faced in evaluation, it represents a valuable tool in the evaluator's toolkit—a set of methods and approaches that, when used judiciously, can enhance the rigor and transparency of evaluation work. Open science has been discussed in, and applied to, a variety of fields, including psychology, medicine, economics, and more (Chiarelli et al. 2019). One question is the extent to which open science is relevant to the field of evaluation. We contend that the similarities between evaluation with research (Wanzer 2021) and the scientific enterprise of evaluation (Patton 2018) suggest that evaluators should be attentive to the open science movement and consider using open science practices in their evaluation work. However, differences between evaluation and research—such as overall purpose, audience, involvement of partners in the evaluation process, dissemination of results, and value judgments (Wanzer 2021)—may also mean that open science might look different for evaluation than it does other scientific fields. One pressing issue in many scientific fields today is the replication crisis, in which a significant number of research studies fail to replicate when repeated. This crisis has been particularly acute in fields like psychology and medicine, such that a combination of methodological issues, publication bias, and questionable research practices (e.g., p-hacking, HARKing) have led to findings that are difficult to reproduce, as described above (Open Science Collaboration, 2015). At the heart of the replication crisis is the concern that many published studies report findings that may be false positives or are not robust, which undermines the credibility of scientific research. In the context of evaluation, these concerns are often particularly relevant. As one example, evaluation often can have political challenges (Azzam et al. 2021), which may exacerbate the prevalence and magnitude of the replication crisis described in other research fields. The inherent nature of working with clients, funders, and commissioners who may have some level of decision-making authority in our work exponentially increases the number of decision points and “forking paths” that we take in our evaluation work and may increase the likelihood of p-hacking, HARKing, and other questionable research practices that limit replicability of evaluation findings. Evaluation partners can also limit the extent to which evaluation products are shared (e.g., plans, materials, data, reports; see Linnell & Tilton, this issue). Thus, the application of open science practices within evaluation could play a key role in addressing these replication concerns. By promoting transparency—through sharing data, analysis code, and materials—open science can help evaluators avoid common pitfalls that lead to irreproducible findings. Furthermore, encouraging preregistration of evaluation plans, sharing methodologies, and opening up the peer review process could help strengthen the rigor of evaluation work, reduce bias, and increase the replicability of findings. We believe that open science practices are beneficial to bring to the field. Open science encourages evaluators to make their data, analysis code, materials, and reports openly accessible. This commitment to transparency translates into increased accountability, ensuring that evaluations can withstand scrutiny in and outside of the field. Additionally, open science practices could foster collaboration among evaluators and others involved in the evaluation process (e.g., funders, commissioners, clients, participants) and provide opportunities for open peer review, adding potential for further improving the quality of evaluation work. It also grants evaluators access to a wealth of resources and studies in open-access formats, promoting informed decision-making and the advancement of evaluation approaches. Furthermore, making evaluation work more accessible and transparent can support evaluation use (see Goodier, this issue). In essence, embracing open science principles discussed above in evaluation practices can enhance the reliability and accessibility of knowledge, leading to better-informed decisions, more effective programs, and increased confidence in the evaluation process. Combined, these unique aspects of evaluation suggest that evaluators need to think critically about the choices they make in their evaluations. Open science practices can help the field of evaluation in mitigating the replication crisis moving forward just as it has been applied to other fields. Open science is a progressive approach to research that prioritizes four key principles: transparency, accessibility, credibility, and reproducibility (Syed 2019). These four principles align with our professional documents in evaluation, including the AEA Guiding Principles (2018b) and the JCSEE Program Evaluation Standards (Yarbrough et al. 2010). Additionally, the open science principle of credibility aligns with the AEA Evaluator Competencies (2018a), and we suggest further updates on the competencies might include more open science competencies. the four principles and how they align with our professional documents in evaluation. with and transparency in to the of the and transparency and sharing of data and findings with the goal of access to information in that and of and should provide to all so and Evaluation should their and data, and our guiding professional documents in evaluation, is also a of evaluation and approaches and et research on evaluation has on evaluation approaches into such as and social and However, a has on evaluation in a variety of aspects that can decision-making in et al. These aspects include methodological and key (i.e., promoting of participation, power might also key related to open science. The key evaluation of power might be relevant et al. for example, evaluation that promote collaborative and decision-making may also be more transparent and with the principles of open science. evaluation might also various open science practices, and future researchers can the alignment of open science with various evaluation For example, one evaluation that may be relevant in about open science evaluation and evaluation is about for and so which to through By on it for across contexts and evaluations. The framework to not just that but it which potential to replication across other through described for of evaluation of these highly align with open science For example, evaluation must have (i.e., specific on how to use the and reproducibility (i.e., whether the can be and This in aligns with reproducibility and replicability of evaluations. an the evaluation data, and could replicate the and to evaluators the in the at the could they to These align with current studies in research, in which many different researchers or research the often to different how researchers have decision points in the research process that can analysis and results (e.g., et al. This various practices within open science, including open data, materials, and open (e.g., open access and preregistration and reports; open open and research evaluation. We also some of the used in open science, including the Open Science For readers interested in to open science, we suggest reading et al. and et al. One of open science is sharing data, materials, and methods more openly for others to for or use in their own work. et al. provide a for promoting transparency in scientific research. how to whether you can you can when you can and how you can on the nature of your research products (see which from et and is shared as part of a article is by et al. which provides for data practices. In evaluation, for example, you can the and the such as in the data of and and data all into this initial decision of whether you can and 2021; et al. 2021). like data on may be more difficult to openly without aspects like of One for open science open as as as 2016). It is also critical to the challenges and associated with these open practices. While the principles and to enhance the of like data and these can be include the and that can limit the accessibility and of research Furthermore, while open science seeks to engage and societal actors more significant to engagement and societal impact. This is particularly relevant in fields where research have social implications but are in that are not accessible to and within open science. and research often and to open science, such as data sharing and open peer this to value and open science practices within and One for sharing resources the Open Science Framework The is a collaboration tool that researchers to make aspects of their accessible for broad including preprints, data, materials, and One of the of this issue has many resources from many of their on the including all of the data, materials, and preregistration for one of their research on evaluation articles (Wanzer 2021; provides a for using to data and other The also has for use the for the first are many other for sharing data, materials, and as (e.g., can use like the for sharing evaluation plans, materials, data, and when Linnell & the extent to which evaluators are sharing evaluation documents and on open science. Furthermore, research on evaluation could their documents Open access making scientific outputs and to scientific This using of the and are a variety of including such as which up and the work into the which the work is to the the which of the work and or but can and the in any or and other can be published in through open access One is open which may include an article to make the article open access the an open access can be a significant for of is open which involves their article on such as the Open Science or other In open the article by the to the can (i.e., the article to an open access to peer and (i.e., the of the article after peer et al. At this are evaluation that are open without article of Program Evaluation and of other evaluation can be published as open access by sharing the or of the article on among other evaluation involves research processes to et al. This can include the research data data analysis and and to be for such as on the Open Science to such as and by a transparent of the which can be the can be in a transparent and researchers can beyond they they in a preregistration in an reports take to the level by of peer the first is after the preregistration and the and the is after the has been and the report and of peer review like the peer review process, with in the field of relevant to the research the preregistration and the reports to provide on the to the In this the of preregistration can as a valuable open science tool for can their evaluation plans, and for in thereby to a specific set of the This parallels the evaluation role in and guiding evaluation by the between and This commitment to transparency the credibility of the evaluation findings and aligns with the core open science principles of transparency and reproducibility. Open science beyond to our evaluation This practices like open peer (e.g., their to et al. 2017), open access (Chiarelli et al. open science (e.g., that articles have open data, open materials, a preregistration et al. 2018), (e.g., the and The and for Open Science et al. provide with a framework for whether and to extent they transparency across research practices (i.e., analysis materials transparency, data transparency, transparency, practices (i.e., and studies (i.e., many For the research practices, can of the practices at one of of shared and and can the to which they with the and use the to practices a to be in a of using the These can beyond into et al. for to use and promote open science practices in evaluations the of many consider replication and of and some and analysis transparency and Open science also into through open resources like or other and resources that are in the or have an open are many evaluation through like Program and Evaluation and Evaluation such as or others by and open such as to Program Evaluation for and the for Program Evaluation in this issue on open science can bring to the field of evaluation, it is to also evaluation can bring to the of open science. is interest in research evaluation (e.g., the Research which on and of scientific research, and open science is one of interest in research evaluation. could be at the forefront of this work, applying the of evaluation and various evaluation to contribute to discussions related to and et al. 2021; 2022), et al. of the and This special issue explores how the described open science principles and practices can be applied to evaluation, and practical for In the open science movement has how research across various disciplines is transparency, accessibility, and The field of evaluation at the intersection of this movement, with unique opportunities to and open science practices. The first article, science practices among of AEA by Linnell and Tilton, results on the current practices and of open science within the evaluation community. The how open science practices, including sharing evaluation plans, materials, data, and are in evaluation, and they the evaluators in embracing these The article, science in impact impact evaluators can from the replication crisis in social by and the that the replication crisis in social psychology offers to impact evaluation. The how the credibility of evaluation can be by that encourage questionable research practices, to in that by from social psychology, impact evaluators can such as preregistration and to enhance their The article, science in and by and the application of open science practices in research. open science has on research, the how evaluation can benefit from data sharing, process transparency, and This article as a for evaluators to open science into practices while article, science for evaluation applying open science principles can further open up the of open science in improving the of evaluation. By applying open science practices like open access and data sharing, that evaluators can increase the and impact of their work, and enhance the broader use of evaluation and Science in from a Program Evaluation for of and a of the evaluation. This article the challenges and benefits of using preregistration in impact evaluation, practical for evaluators to this key open science practice. In data in and challenges of data using open and the critical role of open data in evaluation. a from their the the to open data practices and the that must be when making evaluation data more accessible. and contribute the article sharing and role in improving the quality of open science data sharing which explores how evaluators can enhance the quality and of open science how evaluators can help and to they the of diverse and promote This special issue aims to dialogue and the integration of open science into evaluation practice. The articles the potential for open science to enhance the rigor, transparency, and impact of evaluation through the of various methodologies, challenges, and This issue many evaluation can has been science into its practice. We evaluators reading this issue can at one of open science that they could toward in their own evaluation such as making evaluation products more accessible. As et al. are many decision-making within sharing We also to the that open science is not a one-size-fits-all all-or-nothing are many to open science some of which we with other research fields and some of which are unique to evaluation. we that this issue a shift toward an evaluation with the of open science, in our professional documents (e.g., Guiding Principles for or Program Evaluation of this may lead to to such professional For example, to extent open science be a of Furthermore, despite this issue on evaluation are many for open science should be For example, and commissioners have decision-making power in the evaluation process, and so understanding the extent to which they and promote open science principles and practices are also Evaluation could help evaluation are aware of open science. can other partners in the evaluation process, from partners to the are aware of open science and we may promote open science in our work. the principles of open science the potential to how evaluations are and As we toward this it is for the evaluation to and open science practices. This a shift toward transparency, and accessibility in evaluation as as other interest in evaluation, must engage with open science principles not as a set of but as that evaluation practices.