Supporting fair use in context of instruments in engineering education research: A renewed conversation about validity

London et al. (2021) described engineering education as a “maturing field of scientific inquiry” that lacks the “signposts,” such as theories, techniques, instruments, and methods, characteristic of more mature fields. It is perhaps for this reason that engineering education research has historically relied on instruments developed by scholars outside of engineering education (e.g., psychology, sociology, and science education; Borrego & Streveler, 2015; Malmi et al., 2018). This history has created a body of extant literature with varying levels of consideration made to the unifying concept of validity and the process of validation as an ongoing argument for the fair use of an instrument with the population under study and in the context in which the study occurs. The sheer quantity of studies having overlooked such considerations can easily suggest to new scholars that the approaches of yesterday are still appropriate for today's engineering education research. Douglas and Purzer (2015) brought to the forefront how validity was conceptualized in their editorial written a decade ago. They highlighted the 2014 Standards for Educational and Psychological Testing by the American Psychological Association, American Educational Research Association, and the National Council for Measurement in Education, presented five misconceptions about validity, suggested clarifications to the community to improve assessment-based research, and shared several exemplar assessment instruments for the community to consider. New examples of exemplar assessment instruments have been published in engineering education since, but patterns in publications and submissions wherein authors fail to provide a robust description of the validation process persist. This pattern calls into questions instrument validity, trust in the associated findings, and the ability of scholars to adopt or adapt an instrument for fair use in context for subsequent studies. The discussion for this editorial emerged from experiences across the Editorial Board of the Journal of Engineering Education (JEE). Many submissions continue to omit or overlook modern definitions of validity; in particular, they fail to engage in validation as both a process and an argument. The result is major revisions or rejections for otherwise promising submissions. As an author team, we represent different faculty ranks and roles within JEE, including assistant professor and Assistant Editor, Trevion Henderson; associate professor and Associate Editor, Allison Godwin; and professor, department chair, and Deputy Editor, Adam Carberry. Each of us has experience with instrument development, the editorial process for shepherding manuscripts through the review process, and publishing our own work in this space. We acknowledge that the discussion provided reflects our positionality as US-based scholars and that there are other framings of validity. Our goal for this editorial is to highlight some of the expectations for the publication of instrument development manuscripts in engineering education to better serve the research community and improve the review process for this type of work. Since Douglas and Purzer's (2015) editorial, the definition of validity and standards for evidence in instrument development work have changed. Of particular importance is the definition and standard for validity. The previous standards were a trinitarian view with three distinct facets: (1) content (i.e., Does the instrument measure the domain it is designed to measure?), (2) construct (i.e., Does the instrument accurately reflect the theoretical concept it is intended to measure?), and (3) criterion validity (i.e., Does the instrument predict a real-world outcome that it is designed to measure?; Cronbach & Meehl, 1955). The shift has been to a unitarian view of validity in which multiple sources of evidence are used to support an argument for fair use of the instrument with the population under study and within the context of a particular study (Messick, 1995). This new framing purports that there is not one process for establishing validity, and instruments do not possess the static property of being “valid” or “validated.” Each instance of instrument use in engineering education research should be accompanied by validity evidence for the measures used, which provides a foundational argument that allows consumers of research to better gauge the trustworthiness of the work being presented. Establishing validity is a matter of significant scholarly discussion and debate. For the purposes of discussion, we describe a general framework for building an argument for validity in instrument development manuscripts in engineering education that should not be viewed as prescriptive or an exhaustive list necessary for publication. Instead, we implore authors to integrate the issues noted to help ensure that papers have appropriate validity evidence. There are many forms of evidence widely recognized to support the quality of inferences (e.g., AERA, APA, and NCME, 2014; Campbell & Nehm, 2013; Knekta et al., 2019). A prescriptive approach to establishing validity evidence, wherein guidelines become a “recipe” or “checklist” for publication, should not be taken. Each instrument is different, and the steps chosen to address validation will vary depending on several factors, including the presence or absence of previous validity evidence for the instrument. We do suggest that the field should directly address three core considerations pertaining to instrument development and validation: (1) methodological framework or process of instrument construction, (2) fair use for whom and in what context, and (3) valid inferences. Instrument development papers should describe the methodological framework or process under which the instrument was constructed (e.g., Hinkin, 1998 vs. Kalkbrenner, 2021). The differences between these methodological frameworks are not semantic or cosmetic—they explicate the various research activities involved in the validation process and support methodological transparency (Smagorinsky, 2008). For example, Hinkin (1998) describes the use of exploratory factor analysis (EFA) for item reduction and confirmatory factor analysis (CFA) as key steps in the validation process. Others describe iteration via both qualitative (e.g., interviews and focus groups with expert reviewers), quantitative (e.g., item reduction techniques), and mixed methods approaches (e.g., Creswell & Plano Clark, 2011; Koskey et al., 2018; Zhou, 2019). Authors should make clear the steps involved in the instrument development process when describing the methodological framework. Instrument development papers should discuss the context in which the instrument was developed and employed. This consideration includes the population of the study in both the construction of the measure itself, as well as the appropriate populations for its future use. These discussions should also consider the consequences of instrument use to support the validity principle of fairness. The burden of evidence for the quality of the measurement and its inferences is higher if the measure informs decisions that would dramatically shape student experiences or pathways (e.g., standardized tests, tracking in programs, and admissions). Factors, such as institution mission and type, the nature and structure of the activities under examination, prior participant experiences, and structural diversity (e.g., the relative presence or absence of historically excluded race/ethnicities and genders), may shape the appropriateness of the instrument across contexts. Authors should provide a rich description of the context such that future researchers, guided by the original paper's discussion of appropriate context, might engage in further examination of the validity of the instrument, such as establishing measurement invariance, to ensure the appropriate use of the instrument in their research. Instrument development papers should carefully detail the methods used to gather evidence so that the inferences drawn from instrument scores align with the theoretical reality under investigation, meet statistical expectations for quality measurement, and have practical meaning for use. Authors have a wide variety of qualitative and quantitative methods they may use to support evidence for fair use in context. Each approach entails different affordances, assumptions, and limitations that shape data collection, analysis, and interpretation. Measures of internal validity (e.g., Cronbach's alpha) answer different questions, entail different assumptions, and offer limited evidence for fair use in context compared to methods under the umbrella of Classical Test Theory (e.g., CFA and measurement invariance) and Item Response Theory (e.g., Rasch modeling). Authors should explicate these assumptions and their role in the study design when discussing the evidence for fair use in context. We are not suggesting that any one method is more appropriate, rigorous, or preferable for publication. Authors should carefully explain the affordances, assumptions, and limitations of their chosen approach within the paper's discussion of validity to support inferences by explaining methodological decisions. We argue that alignment of the issues discussed supports fair use in future research, as well as reproducibility across contexts. We encourage the engineering education community to engage with this work as it provides a valuable discussion of validity that has not yet been taken up broadly within the field of engineering education. To assist, we provide a table of reflective questions with example references in the Appendix for the community to engage with in the production of scholarship related to measurement and the use of instruments. Gentry et al., 2023 Verdín, 2025 Verdín et al., 2021 DeMonbrun et al., 2017 Hammer et al., 2024 Douglas et al., 2020 Lee et al., 2022 Mamaril et al., 2016 Trevion Henderson is assistant professor of Mechanical Engineering and STEM Education at Tufts University, where he also serves on the Steering Committee for the Institute for Research on Learning and Instruction and as a Faculty Fellow for the Center for Engineering Education and Outreach, 200 College Avenue, Medford, MA, 02155, USA. Email: [email protected]. Allison Godwin is the Dr. G Stephen Irwin ’67, ’68 Professor in Engineering Education Research and an associate professor in the Robert Frederick Smith School of Chemical and Biomolecular Engineering at Cornell University, 120 Olin Hall, 113 Ho Plaza, Ithaca, NY, 14853, USA. Email: [email protected]. Adam Carberry is Professor and Chair in the Department of Engineering Education at The Ohio State University. Smith Laboratory, 174 W 18th Avenue, Columbus, OH, 43210-1168, USA. Email: [email protected].