Commentary: Past, Present, and Future of Educational Measurement

As the task force seeks to find consensus around the foundational competencies in our field, it is appropriate to take some time and critically examine the process by which the task force created this document and the views that it represents. Overall, our reaction is that the professional who was trained according to this consensus document will not be fully capable of participating in the testing industry as the administrator of policies and techniques that can sustain our field and profession in the long term. We want to start by acknowledging the context in which the focal article and this comment are written, aligned with the inclusion of Subdomain A: Context: social, cultural, historical, and political, now applied to this commentary piece. We live in the years after the postpandemic, situated in the context of the United States, with relevant historical events happening then. As for what is immediately related to Educational Assessment, we have seen disruptive events in our industry and profession. The COVID-19 pandemic created significant shifts in how test users relate to educational testing, extending their use of tests and at the same time limiting the use of tests in some contexts. Before the beginning of the pandemic, there was an increase in criticism of Standardized Testing. A major user of College Admissions Tests (UC California) discontinued their use, and a president of American Educational Research Association (AERA) directly criticized the use of Standardized Testing for Accountability purposes. The pandemic proved to be an opportunity for many prior users of tests in the college admission process to switch to the optional use of testing, with similar changes in graduate-level requirements. Moreover, we saw another big disruptive event in our field in recent years: Generative AI and Large Language Models. The availability of over-the-counter LLM is creating immense opportunities and challenges. Test security faces new challenges with the incorporation of those technologies into operational use, as many methods of cheating are hard or impossible to defend against unless we make significant changes and adaptations in our policies and practices to maintain test security. LLMs are also opening the door to multiple uses in our field: Automatic Item Generation, Automatic Essay Scoring, Automatic Item Template Generation, among others. We live in a time of rapid change for our field as is the case for many others. Our main concern with the document provided by the task force is how it relates to our field's future. On the one hand, the performativity of the document (how will different audiences read and interpret the document, and will it impact those audiences toward change and action; Cavanaugh, 2015) seems not to have received sufficient attention as if the authors were unaware or chose not to address its own relevance and future implications. What we describe as a consensus document for the foundational competencies in our field will shape that future. How intentional was the task force in wanting to shape this future? To the extent the document has an intention, this seems to be for testing professionals to continue their activities as they are currently performing them rather than disrupting current practices. In our view, the document is not disruptive enough. A few aspects of the document seem attuned to present challenges; however, there needs to be a better developed view of the future of the field. In the existing document, to the extent that there is a comprehensive view it is implicit and tends to be conservative. A clear connection with the past, an evaluation of the present, and a communal construction for a view of the future seem “foundational” for such a document. Educational measurement involves the measurement of knowledge, skills, dispositions, and abilities for some educational purpose, such as supporting learning, certifying learning, or identifying policies and practices that improve learning. A career in educational measurement is conceived broadly as work that supports the design, use, and evaluation of measures of cognitive, affective, and other psychological constructs developed in educational, training, and learning environments. (p. 6) What competencies do we leave behind because of this definition? As of now, the document does not include anything related to qualitative research, for example. The foundational competencies do not even include ethical aspects of our profession, like legal aspects of the context or our own codes of conduct as a professional organization (Schmeiser et al., 1995). How consequential is it to leave some competencies out of our jurisdiction? So, in the spirit of connecting with the past and look to the future, we should ask ourselves: how a practitioner taught using the foundational competencies would have behaved in the situations mentioned by Sireci and Randall (2021) about the history of Fairness? Difficult to know for sure, but without ethics in what is foundational, we are exposed to repeat our most terrible mistakes as a field. We take the position that, first and foremost, the document should imagine a technique that is sustainable over time. Pushing for a sustainable technique means, in the end, that we include what the task force mentions as foundational competencies, plus anything else necessary for the future professionals of our field to coexist in a world in which our technique does not cause harm, coherent with a project of social justice. This stance is our personal view of what is sustainable. We need to debate more about what sustainable means in our field, how the technique of testing can be effective and capable of sustaining itself over time. As of now, such a view about the future of testing is either missing or not made explicit in the document. The procedure for consensus creation for this document has some important limitations. The process resembles what we usually have done for the Standards for Educational and Psychological Testing (AERA, APA & NCME, 2014; from now mentioned as the Standards), in which the writers accept rounds for commentary pieces during and after the writing process, but no actual democratic procedure is performed. It is interesting to see how the first domain Communication and Collaboration was written. The section highlights the importance of dialogue, and yet the domain is focused on how, we the ones possessing the technical knowledge, should communicate with others. There is no communication from those others to us. This is a pattern in our field, in which decision making is legitimized by the expertise of the participants. This way of creating consensus (about our own topics of debate, and the ones with the public about testing) is politically weak, and potentially dangerous. Part of the development of a sustainable technique will likely require us to explore different ways to generate consensus among ourselves and democratize the way we construct and justify the uses of tests. Who is affected by the use and existence of a consensus around the foundational competencies in our field? Who should have a voice and vote in that decision? How are we replicating the structures of power and oppression that are part of our context? Are we disrupting that logic or continuing it? What counter-narratives were generated around the document? Do we have documentation about differing views that did not get into the document? What were those divergent views? The document itself and what it defends are insufficient for practitioners to respond to these questions in the future, for the document about foundational competencies and for any test administered by a professional of our field taught under the principles pointed out by the document. It is not appropriate to use expert knowledge to undermine the self-determination of the communities impacted by testing. Political problems will not disappear because we do not talk about them. The way we generate consensus, the way we relate to qualitative research methods, the way we relate with values, all the previous are part of a political dimension of our profession that is hard to evade. After all, we are working with human beings, taking decisions about their lives (even sometimes about their bodies). How could our profession not be political? The focal document, its construction procedure, is continuing a different view about measurement and politics, in which science is protected in a kind of “ivory tower,” isolated from external influences and independent from moral judgments (Borsboom & Wijsen, 2016). We see a contradiction in that the document, in Subdomain 3.A, includes what is political, but this is not reflected in how the document itself was written. So far, the task force has tried to represent a diversity of stances, but without demonstrating self-awareness or the disclosure of differences in perspectives of the members of the drafting group, and no responses to critical questions (see Boveda & Annamma, 2023 for examples for those questions). The way the task force was constituted also does not build a basis for legitimacy of the document by the use of democratic mechanisms. This is a missed opportunity and a conservative stance relative to the future of our field and about the way we construct consensus. An alternative way of doing the consensus generation is the one the Graduate Student Issues committee used to develop a statement about the new edition of the Standards (Araneda et al., 2023). We will now talk about some other observations for the document as part of the commentary piece, providing more discussion for each. We will conclude with a summary of the commentary piece and some observations for the future of our field. It is interesting to evaluate the competencies from the lens of test security. How well prepared will a student in our field be to understand the problems related to this specific subfield? In general, Test Security is a Fairness problem. Right now, the foundational competencies do not include test security. We see a general lack of consideration for the more concrete problems that emerge from using new technologies in educational testing across the foundational competencies in its current version. Test Security could be part of Subdomains 3.B and 3.C. The Fairness section should mention how test security is part of the fairness problem and, in Theory and Instrumentation, how essential requirements in educational testing like test security rely on considerations included since its early design and development phases. The task force did not include our field's ethics in the current version of the foundational competencies. The task force included Fairness as part of the document; however, Fairness is not the whole of our field's ethics. If you look for the word “ethics” in the document, you will not find any match on it. At the very least, we as professionals should be aware of the legal boundaries of our profession. We can include this in Subdomain A, as the legal context. For it, students could learn about the existence of different types of Standards, legal cases associated with using tests in the United States, the Code of Professional Responsibilities in Educational Measurement (Schmeiser et al., 1995), and other codes of ethics in relevant fields. This section can include the classic ethical considerations for research with humans, IRB process, and other related. The ethical dimension should be foundational. By not including it, we can create the impression in the document that an amoral type of practice, in which normative aspects of using testing as a technique are disregarded, is acceptable. An alternative view of our future is to put the normative aspects at the forefront in the use of testing as a technique. One need we see in the foundational competencies is for them to encourage psychometricians to master more mathematical models and be ready to see psychometrics as a mathematical modeling task rather than an expertise in a few specific mathematical models. Following Domain 1, we can rely on others if we want to use models different from the classical ones. Nevertheless, we need to know what other methods, models, and mathematics exist for that collaboration to happen. This observation also has some historical connotations. We walk on the shoulders of the giants of our field: Galton, Thurstone, Lord, Cronbach, and so many others. The spirit that led them to create the models we use now came from a will to model and be scientific about psychological and educational phenomena, of humans being studied and interpreted. We accept as almost sacred some methods. This was certainly not the intention of the creators of these methods. We should be the heirs of the spirit that led the designers to create the models we now use rather than sticking to a few models just because we know them well. This is a specific way of looking at the future of our field, in which we continuously try to expand the nature and reach of the types of models we use. This document did not have any consultation with graduate students at NCME. In consultation with the chairs of the GSIC during the time in which this first version of the document was written, they do not recall any instance in which the task force invited us or even mentioned this process. The goal of the task force and the intention of the NCME Board to be explicit about what is foundational for our field is a step forward. We appreciate the beginning of this discussion, which will continue in the future. We need to acknowledge and show our appreciation for the big efforts and professionalism of the members of the task force, but we also need to recognize that their contribution has been limited by the historical blind spots of our field. It has room for improvement in its performativity and its political dimension; it misses meaningful points in the discussion over the years and has a conservative view for the future. Sadly, the limitations in the document will be part of the reputational debt the older generations are leaving to the new professionals entering the field. We need this and all other consensus documents in which our organization participates to reflect on our past, our present, and the future we can build. We hope new versions of this consensus about the foundational competencies in educational measurement can be written, and new ways of building consensus can be explored, that can own better its political dimension, be representative, and more attuned to the context in which our field exists.