Commentary: How Research and Testing Companies can Support Early‐Career Measurement Professionals

The National Council on Measurement in Education (NCME) Presidential Task Force on Foundational Competencies in Educational Measurement (which we refer to as the Task Force) has performed an important and overdue service to the field. The Task Force conducted a remarkably thorough analysis and produced a framework and report that offer relevant, actionable guidance. Because the report is so well-conceptualized and is likely to exert a strong influence on the field, we appreciate the opportunity to share some reactions and ideas for ways to strengthen its relevance and impact. In particular, and in light of our current and past roles in research and industry, we were pleased that the Task Force offered clear guidance on graduate program curricula while acknowledging that much professional learning occurs outside of those formal programs. Our primary rationale for writing this commentary is to expand the conversation around how learning occurs in two types of workplaces—testing companies and research institutions outside of the traditional higher education sector. We draw on our own perspectives as employees, interns, and mentors at several such organizations. Our objectives are threefold. First, we hope our guidance for graduate programs will help ensure their students are prepared for a wide variety of professional opportunities. Second, we aim to encourage our colleagues in research institutions and testing companies to embrace their responsibility for supporting the lifelong learning of measurement professionals. Finally, we hope to play at least a small role in informing the ongoing conversations at NCME and in other professional organizations regarding how we can best ensure that future generations of measurement professionals are prepared to tackle the growing challenges that the field is facing. We bring different lived experiences and are at different stages of our careers, but we share a commitment to supporting the development of early-career measurement professionals. In what follows, we highlight a few features of the Task Force report that we found especially noteworthy and valuable based on our professional experiences. We then identify some areas in which we think the report could have gone further to inform graduate education, and we present recommendations to colleagues in research and testing organizations regarding how they can support the development of early-career measurement professionals. We found a lot to like in the report and framework. In this section, we highlight a few features that we think are especially relevant to the preparation of professionals for careers in testing and research outside of academia. The initial definitions in the framework set the stage for what was to come: “An educational measurement professional is a learner who is employed in an educational measurement career” (Ackerman et al., 2024, p. 14; emphasis in original). From the outset, the authors acknowledge the importance of lifelong learning. They point out that a degree is but the beginning of a career of self-improvement. Especially in industry, this attitude is necessary for success, as the challenges faced by testing and research organizations continue to evolve. Accordingly, Domain 2—Technical, Statistical, and Computational Competencies—stresses not just learning how to apply methods, but also the understanding of their logic and purposes (p. 10). Otherwise put, it is not sufficient to be a technician, uncritically applying learned techniques. A successful learner will develop the skills to be an engineer, who “can develop, adapt, and evaluate statistical models and computational algorithms for educational measurement applications” (p. 11). While a technician will be at a loss when a novel situation arises, an engineer can improvise, adapt, and overcome—something all three of us have found to be critical in our professional roles. We were pleased to see that Domain 3—Educational Measurement Competencies—included a subdomain on context competencies. These social, cultural, historical, and political contexts influence how individuals interact with measurement instruments. As importantly, context influences how others interpret the results of those measurements. Understanding more fully the lenses through which we develop, administer, complete, and interpret measurements can push us closer to developing measurements that are as free from bias as possible and that allow individuals to see themselves accurately reflected in the assessments and the results. Domain 3 also includes a subdomain on validity and fairness. Testing professionals need to be aware of the importance of evaluating whether uses of test scores are justified by the evidence. Professionals must ensure that the justifications hold to the same degree for all groups of interest. Although it is not stated in the report, we believe that the intersection of this subdomain with the context subdomain will necessarily lead future measurement professionals toward a more individualized validation, closely aligned with Mislevy's (2018) conceptualization of conditional fairness. Returning to the initial notion of lifelong learning, we appreciate the emphasis in the framework on graduate school training as a starting point. The authors stress the importance of job training and continued professional development. As the authors state, “Applied measurement rarely conforms to theoretical models and idealized assumptions” (p. 16). Thus, the authors point out, professionals often need to combine several competencies to find solutions in the presence of real-world constraints, optimizing the most important features of a measurement situation while minimizing harm. Clearly the framework resonated with us, and we especially appreciated its applicability to a variety of career paths. As we considered the guidance offered to graduate programs, we identified a few places where we think some elaboration of the guidance could be helpful, especially from the perspective of professionals working in industry and nonacademic research institutions. Importantly, we recognize that this perspective is limited by our own experiences, including the specific institutions in which we have worked and learned, and that our observations will not generalize to all contexts. Theories, textbook examples, and artificial datasets provide crucial but limited opportunities for learning. All three of us fondly recall our first exposure to certain theoretical frameworks, and many of our required course readings continue to resonate. Similarly, learning how to program and conduct sound statistical analysis, whether in Minitab, C, R, or Python, enabled us to move into professional roles smoothly. However, we all experienced early on-the-job learning that clarified the ways in which theories and data often misbehave in the real world. These experiences helped us identify missed opportunities for training to integrate academic work more closely with applied, real-world activities. We appreciate that the Task Force emphasized “co-curricular activities” as an important part of graduate training, and we concur strongly. Without the opportunity to apply skills to real-world, on-the-job situations, learners will likely begin their careers at a disadvantage and will need to adjust their perspective from learning theories as applied in a narrow context to the broad range in which skills can be applied on the job. As an example, our first author, who is currently finishing his dissertation, developed critical knowledge and skills through graduate coursework but was not exposed to the wide range of use cases and activities (e.g., FAFSA/high school completion, professional development) until participating in an internship and then full-time employment. Unfortunately, such opportunities are not available to all students. Expansion of partnerships between graduate training programs and research and testing organizations could expand these opportunities, especially if they prioritize collaborations with graduate programs that have traditionally lacked resources and connections to such experiences. An important skill is the ability to design and conduct principled simulations. Domain 2 includes the ability to conduct simulations under technical competencies. Too often, though, simulations we have seen include unrealistic conditions that render their results unusable. We believe that to be useful, simulations must match as closely as possible to real-world phenomena. Walker (2012) offered a set of principles that might lead to more useful simulations. Briefly, the simulation model should (1) be clearly stated and results replicable; (2) fit real data; (3) yield falsifiable predictions; and (4) be able to generate data over a wide range so as to allow discovery of boundary conditions, but without exceeding the realm of possibility. Mostly, Walker cautions that researchers remember that a simulation is construction, not truth. We would like to see more elaboration on this issue in the framework (and, more importantly, in graduate programs). Intellectual humility is often underemphasized. The committee included intellectual humility in its list of dispositions and mindsets described in Guiding Principle 6. Porter et al. (2022) state that “Intellectual humility involves recognizing that there are gaps in one's knowledge and that one's current beliefs might be incorrect” (p. 524). Especially in today's polarized media and political environment (Woo et al., 2024), graduate programs must guide students toward an understanding that everyone, from a novice graduate student to the most heralded leader in the field, can learn from others in ways that might upend their long-held views. Similarly, programs should encourage students to build skepticism of what they are taught, even if that teaching is coming from a preeminent, revered professor or presented in a textbook that has graced the shelves of faculty members for decades. Students benefit from courses in a variety of disciplines. We agree with the Task Force that educational measurement competencies “overlap and interact with competencies in other professions and disciplines” (p. 1). Some of the most gifted psychometricians we have known did not have degrees in educational measurement, but in psychology, mathematics, and related fields. We believe much of their success stemmed from their ability to view problems from a different perspective than everyone else. As an example, one of us remembers interviewing a candidate for a psychometrician position. When asked to recall having to solve a particularly difficult problem, the candidate described her frustration with trying to find a satisfactory commercially available item selection tool for a computer adaptive test. Because of her background in a different industry, the candidate said that she was able to recognize the desired functionality in a piece of software used in manufacturing. We can think of many success stories that have come from such diversity of thought. For this reason, we believe that measurement programs would do well to encourage learners to explore other disciplines: quantitative disciplines like mathematics and statistics, for sure, but also, philosophy, music—anything that might open their minds to possibilities. Knowledge of other disciplines may also give learners and professionals better insight on applications of educational measurement for various samples and use cases. If we were to add a course to the curriculum outlined by the task force, it would be in ethics. The Joint Standards, the APA Guidelines for Psychological Assessment and Evaluation, and readings on social justice serve as a critical foundation for the work conducted by educational measurement professionals. As society evolves, the ability to view the world, including measurement situations, through an ethical lens, becomes more critical (Lorié, 2024). Ethical thinking will foster early-career professionals’ commitment to considering and prioritizing the rights and dignity of the individual. Measurement can then become a collaborative process in which all concerned parties share power across all stages of the process (Walker et al., 2023). Applications of educational measurement beyond the individual level are underemphasized in the Task Force report and in graduate training. The Task Force wrote that “A career in educational measurement is conceived broadly as work that supports the design, use, and evaluation of measures of cognitive, affective, and other psychological constructs developed in educational, training, and learning environments” (p.6). Based on our experiences in industry and research, we worry that an emphasis on psychological constructs misses opportunities to prepare measurement professionals who can apply their expertise at the group or institution level. Such expertise is especially important as the field aims to counter deficit narratives in reporting on achievement test scores, in part by incorporating high-quality measures of the learning environments that contributed to those scores (Marion, 2020; NASEM, 2019). Along similar lines, efforts to mitigate systemic racism and other social phenomena that contribute to inequitable opportunities would benefit from better measures of organizational and social conditions. We agree with the Task Force that preparation for careers in educational measurement begins during graduate training and extends into early-career activities, with the aim of laying a foundation for lifelong learning. In the previous section, we described some ways in which graduate training might be adapted to improve the preparation of graduates. In this section, we offer recommendations for employers of early-career measurement professionals. We focus on testing and research organizations, reflecting both our own expertise and the significant role that these institutions play in advancing high-quality, equitable educational measurement. Our overarching recommendation is for employers to create the conditions and learning opportunities to promote on-the-job development of a broad set of competencies. Guiding principle 6 described several dispositions and mindsets such as critical thinking, creativity, and flexibility that are sometimes called social and emotional competencies, soft skills, durable skills, or employability competencies, among other terms (Anderson, 2018; CASEL, undated; Pelosse, 2022). Importantly, research demonstrates that “The brain and development are malleable across the entire life span in response to relationships, experiences, and contexts” (Darling-Hammond et al., 2022). Early-career professionals will continue to hone these competencies through various on-the-job experiences, and employers should strive to design these experiences in ways that support professional growth while meeting client needs. An underappreciated factor contributing to this development is the modeling of the colleagues who mentor and collaborate with early-career professionals. As the field aims to tackle significant challenges and areas of disagreement related to applications of measurement, we all must be able to engage in respectful debates, be open to having our views challenged, and be committed to actions that are ethical and that benefit society, even when doing so is extremely difficult. We offer the following recommendations based on our own experiences and our interactions with large numbers of graduate students, interns, and early-career employees. Promote the art of active listening. The Task Force's Domain 1—Communication and Collaboration—includes “cocreating solutions in ways that synthesize or build on ideas offered by other team members. They also include the ability to understand a variety of perspectives, manage priorities of all participants, and meet expectations as a member of a team” (p. 3). Learners at any stage in their careers must be active listeners who consider others’ perspectives while applying a critical lens to measurement theories and in which contexts they apply. They will also need to be able to listen actively to client requests and solutions from team members to perform job tasks, including selecting and applying appropriate measurement theories and statistical methods. Encourage thoughtful skepticism. Colleagues should model and invite new professionals to question decisions rather than simply following orders from someone higher in the corporate organization chart. New employees might naturally worry about rocking the boat or being labeled as “difficult,” and these concerns can be especially salient for professionals from traditionally marginalized groups. Employers should create a culture that encourages and protects constructive debate, ensuring that new colleagues feel a sense of safety and know that their views are respected. Early-career professionals should also be able to think critically and skeptically about what they learned in graduate school, such as the history and application of theories and methods; and they should be able to interrogate the application of existing theories and methods in novel situations and use cases. The framework mentions that “just as advances in learning theories may lead to new approaches and methods of instrumentation, advances in instrumentation may lead to novel insights that lead to changes in learning theories” (p. 5). Workplaces should adopt the conditions and supports to help new colleagues adapt to novel situations when currently accepted theories and methodologies do not fit. Nurture a commitment to advocacy within the constraints of the institution's mission and values. Guiding principle 4 states that competencies in educational measurement should be “aspirational about the future of the profession and discipline” (p. 1). Such aspiration requires professionals to serve as advocates for the communities who use or are affected by the assessments we create. In advocating for these communities and advancing the field, professionals must also be aware of the history of educational measurement within various communities, as described in Domain 3, Subdomain A: Context: social, cultural, historical, and political. Educational measurement has not been utilized in beneficial ways, particularly in marginalized communities. a focus on equitable measurement that opportunities for all educational measurement professionals should also for theories and methods that are appropriate for use and that into the contexts in which can or should apply. the can concerns in institutions that are committed to so it to be and in with the institution's a sense of The Task Force that the history of the field the of to and and others have out the ways in which educational measurement has contributed to inequitable and and has to the field to a of professionals is a crucial in it in a more beneficial and educational internship programs, and organizations such as in Measurement have significant to improve the diversity of the to those efforts is the that for careers in measurement have experienced of testing in their own or with their or communities. Such experiences might from their sense of safety or in the field. Employers can help a sense of through such as groups and collaborations with but these formal are likely to it is responsibility to interact with colleagues in ways that the that they not in that institution but that they play a crucial role in it and employers should support in this and guidance to help early-career professionals adapt to new experiences. Domain 2—Technical, Statistical, and Computational a variety of statistical and research methods and activities (e.g., and learners will likely perform in educational measurement Because data we on the job do not theoretical and methods, new professionals should be the to interact with or in with more experienced colleagues who can provide appropriate guidance and Similarly, even when graduate students have opportunities to with they are often from some of the such as with working with or even out a A mentor or who is available to support new colleagues while also as a role model can have a on new to the with graduate programs. learners can how their knowledge to on real-world Unfortunately, there are not for student in a measurement and testing organizations can expand their influence by with graduate programs to find ways to engage students. Professionals from these organizations could offer or in the and they could students in problems of from Such activities could include partnerships with institutions to diversity in research and testing organizations. The Task Force report has a for the measurement field to what it to develop and support early-career measurement professionals. our reactions and we hope to contribute in some small to a new of professionals who will be to meet this in the history of educational measurement. We are especially that graduate students and early-career professionals will have strong and significant in these and we encourage our colleagues across the field to embrace our responsibility to the members of our We have known of to