Still Interested in Multidimensional Item Response Theory Modeling? Here Are Some Thoughts on How to Make It Work in Practice

Given tremendous improvements over the past three to four decades in the computational methods and computer technologies needed to estimate the parameters for higher dimensionality models (Cai, 2010a, 2010b, 2017), we might expect that MIRT would by now be a widely used array of models and psychometric software tools being used operationally in many educational assessment settings. Perhaps one of the few areas where MIRT has helped practitioners is in the area of understanding Differential Item Functioning (DIF) (Ackerman & Ma, 2024; Camilli, 1992; Shealy & Stout, 1993). Nevertheless, the expectation has not been met nor do there seem to be many operational initiatives to change the status quo. Some research psychometricians might lament the lack of large-scale applications of MIRT in the field of educational assessment. However, the simple fact is that MIRT has not lived up to its early expectations nor its potential due to several barriers. Following a discussion of test purpose and metric design issues in the next section, we will examine some of the barriers associated with these topics and provide suggestions for overcoming or completely avoiding them. Tests developed for one purpose are rarely of much utility for another purpose. For example, professional certification and licensure tests designed to optimize pass-fail classifications are often not very useful for reporting scores across a large proficiency range—at least not unless the tests are extremely long. Summative, and most interim assessments used in K–12 education, are usually designed to produce reliable total-test scores. The resulting scale scores are summarized as descriptive statistical aggregations of scale scores or other functions of the scores such as classifying students in ordered achievement levels (e.g., Below Basic, Basic, Proficient, Advanced), or in modeling student growth in a subject area as part of an educational accountability system. Some commercially available online “interim” assessments provide limited progress-oriented scores and subscores from on-demand tests. However, the defensible formative utility of most interim assessments remains limited because test development and psychometric analytics follow the summative assessment test design and development paradigm: focusing on maintaining vertically aligned or equated, unidimensional scores scales (e.g., a K–12 math scale). The requisite test design and development frameworks for summative tests focus on the relationships between the item responses and the total test score scale (e.g., maximizing item-total score correlations and the conditional reliability within prioritized regions of that score scale). Applying MIRT models to most summative or interim assessments makes little sense. The problem is that we continue to allow policymakers to make claims about score interpretations that are not supported by the test or scale design. The standards on most K–12 tests are not multidimensional. Rather, they are a taxonomy of unordered statements—many of which cannot be measured by typical test items—that vaguely reflect the intended scope of an assessment. Some work is underway to provide “assessment standards” that reflect an ordered set of proficiency claims and associated evidence (measurement information) that changes in complexity. The reported scores may be viewed as composite measures representing two or more content domains or subdomains. But they still tend to function as a unidimensional scale. A unidimensional composite can be a mixture of multiple subdomains or content areas as long as the underlying, unitary trait can be empirically demonstrated to satisfy local independence under a particular IRT model. Most MIRT studies applied to summative and interim tests are just exploratory factor analyses. That is, the models may help isolate minor amounts of nuanced multidimensionality and researchers may then attempt to interpret the patterns of residual covariance in some content-focused way. However, whenever we develop and select items with high item-total score correlations (e.g., point-biserial correlations), we build our tests to provide a singular measurement signal—an essentially unidimensional scale. We might pretend that we can legitimately organize items into content-based strands and report subscores. However, the subscore item groupings tend to be statistically unjustified and merely result in less reliable estimates of the (essentially) unidimensional trait supported by the data (Haberman & Sinharay, 2010). The point is that subscores—any subscore on an essentially unidimensional test—should NOT be computed nor reported. Developing reliable, valid, and useful subscore profiles demands a commitment to designing and maintaining multiple scales. Instead, consider a different and perhaps more useful formative assessment purpose—at least potentially useful to teachers, parents, and importantly, the students. While some conceptualize formative assessments as low-stakes classroom assessments, the critical value of these assessments for improving instruction and changing student learning in a positive way cannot be minimized. Well-designed formative assessments should arguably be based on multiple metrics that are demonstrably sensitive to good instruction, curricular design, and student learning. They further need to be offered on-demand—possibly daily—and provide immediate or at least timely, detailed, and pedagogically, actionable information to teachers. From a test design and development perspective, the implication is that formative assessments must provide useful and informative performance profiles for individual students that reliably identify valid student strengths on which to build and weaknesses to remediate, as well as simultaneously monitoring progress concerning multiple traits or competency-based metrics. The central uses of formative assessments align well with the capabilities of MIRT modeling where the latter provides numerous technical psychometric tools for building and maintaining multiple score scales. [Note: this statement extends to diagnostic classification models (DCMs) where discrete, ordered traits or attributes replace the continuous proficiency metrics assumed for most MIRT models. See, for example, Sessoms and Henson (2018). One promising new approach for diagnostic DCM is by Stout et al. (2023).] The challenge is that adopting a MIRT model or DCM is not, in and of itself, a formative assessment solution. A different test design and development paradigm is needed. At this juncture, it seems important to remind ourselves that we fit psychometric models to data—not the reverse. Therefore, the important issues in this article are not centered on which MIRT or DCM model to use or which statistical parameter estimators to employ. Those are psychometric calibration and scaling choices. The most important issues revolve around the characteristics of the data—starting with how we can efficiently design and create items, and then assemble test forms that can meet our intended formative assessment information needs. Our extensive experience with, and research-based knowledge from, large-scale summative testing may not apply to formative assessment systems. For example, we need to consider different mechanisms for evaluating item quality, calibrating the items, linking or equating scales, and scoring student performances. If we can agree on the utility of formative assessments and the ensuing need for multiple constructs, the obvious question becomes, “Which constructs should we measure?” It is not sufficient to write test questions to nebulous content and/or cognitive specifications associated with content-based subdomains, factor analyze the results from large-scale field trials, and then play the “Name that Factor” game. Each scale needs to have a concrete purpose supported by its design properties and development priorities. Figure 1 displays the four primary domains from the Common Core State Standards (CCSS) for Grade 2 Mathematics (NGA & CCSSO, 2010). Additional detail is provided at the Clusters and Standards levels for the Measurement & Data domain. Now contemplate this CCSS example from the perspective of a formative assessment design. At the very least, we would need four score scales, one for each domain (2.OA, 2.NBT, 2.MD, and 2.G). While likely to be positively correlated with one another, it seems implausible that most second-grade students would have the same educational opportunities to learn and almost identical levels of mathematics knowledge and skills—or even highly consistent patterns of performance across these four domain-based proficiencies. Often, dimensionality is correlated with the placement of students with learning. Before receiving instruction or long after mastering the material, data appear unidimensional. It is only when students are actively challenged by learning that dimensions appear. A well-designed formative assessment system might expect to observe different score patterns to emerge across the four domains reflecting different patterns of student strengths and weaknesses concerning the domain-specific knowledge and skills measured. Consider Figure 2. The left side of the figure depicts the intended structure. That is, the four ellipses are the constructs with the curved connectors denoting nonzero covariances among the scales. The middle image shows the potential magnitude of the six covariances between the traits—which would be proportional to the cosines of the angles between each domain-based scale. From a measurement perspective, each trait is a factor or reference composite (Luecht & Miller, 1992; Wang, 1986) that psychometrically functions as a unique scale. Finally, the right side shows the score profiles for three students. This figure outlines a high-level target formative assessment scale design! Figure 3 explicitly shows more detail about how our test and scale design goals are substantially different for unidimensional (summative or interim) and formative assessments. Under a unidimensional design paradigm (left side of Figure 3) all items within each of the included content domains are expected to be highly correlated with one another (+++) and with all items in other domains. Conversely, for a formative test designed to measure four rather distinct domains (right side of Figure 2). The items would correlate well within a domain but not as highly across domains. These correlational patterns are intentional design goals. They are not accidental nor are they likely to be “discovered” via creative factor analyses. We need to build our items and the scales synchronously so that statistical structural modeling confirms the intended scale structures and properties of the test items. If we subsequently find that our target structures are not being met, we will likely need to impose serious item design constraints on the cognitive complexity of the items within each domain to deemphasize and isolate the intended signals for our constructs. The nature of the (intended) multidimensionality underlying may also change over time as shown in Figure 4. This figure provides an example of a high-level scale-design requirement that might support a domain-specific formative assessment system to be used during a single semester or entire academic year. Some foundational knowledge and skill domains may become more prominent early in the semester and disappear or be absorbed later on. See Luecht (2003) for a substantive example involving the changing dimensionality of oral language proficiency. Figure 4 shows four epochs (e.g., time periods that could correspond to different learning modules or the scope and sequence of learning objectives). There are also five measured traits or constructs (θ1, θ2,…,θ5). The structural diagrams near the top of each rectangle denote the constructs and their inter-relationships (i.e., covariances). The double-ended arrows denote the nature of those covariances where smaller angles correspond with higher covariances or correlations, proportional to the cosines of the angles. Figure 4 suggests that we need two scales for Epoch #1 θ1 and θ2. We transition to three scales to profile student strengths and weaknesses at Epoch #2, where θ1 and θ2 become more highly correlated and a third scale, θ3, is needed. (Note that the rationale for the increased magnitude of the correlation between θ1 and θ2 as we move from Epoch #1 to #2 is, as students learn and simultaneously master two or more traits, we would expect the inter-trait correlations to increase toward a unidimensional composite.) At Epoch #3, θ1 drops out of the intended multidimensional score profile and θ4 emerges—with θ2 and θ3 now becoming more correlated with one another. Finally, by Epoch #4, θ1, θ2, and θ3 coalesce, and a fifth trait, θ5, is added to the measurement profile. The type of dynamic score-scale profile design implied by Figure 4 is unlikely to happen by starting with an item bank constructed to primarily support a unidimensional score scale (e.g., a summative test item bank or a simply a bank where items are screened have the highest possible item-total test score correlations). Building and maintaining multiple useful scales to support robust score profiles starts with an in-depth understanding of the nature of each construct at a very detailed level. We are essentially building multiple test batteries (e.g., a battery for each epoch shown in Figure 4). This is not a trivial undertaking but, as discussed further on, is feasible using a principled scale and item design framework. The test design and development challenges for building and maintaining multidimensional score scales to support formative assessment are not trivial. There can be substantial up-front costs to engineer a robust infrastructure that can: (a) generate potentially massive item banks: (b) support the intended multidimensional score scale structure(s); (c) significantly reduce or eliminate pilot-testing and calibration of individual items; and (d) support optimal within-domain test assembly constraints and forms construction for on-demand testing, possible with multiple epochs (Figure 4). The longer-term payoffs emerge from integrating multidimensional modeling efforts with a robust architecture for generating isomorphic items to detailed task model specifications. Following the Assessment Engineering framework, Luecht and Burke (2020) proposed a test design and development paradigm that concentrates scale design on manipulable properties of the tasks (also see Luecht et al., 2010; Luecht, 2012a, 2012b, 2013, 2016). Assessment Engineering advocates for laying out in a detailed fashion the ordered proficiency claims for each scale. For example, foundational claims about knowledge and skills are superseded by incrementally more rigorous claims. The core Assessment Engineering item and test development technology is called task modeling. Task modeling focuses on engineering two critical aspects of instrument design. First, it incorporates item-difficulty modeling research to establish empirically verified complexity design layers that control the cognitive complexity and difficulty of entire families of items called task-model families. Note that discrimination is NOT something that we care about directly with item difficulty modeling. By controlling cognitive complexity via the complexity design layers, we can statistically locate each task-model family on the underlying scale and further use them as quality control indicators for new items as they are generated for each family. Items within each task-model family are treated as statistically and substantively isomorphic (i.e., exchangeable use for scoring and score interpretation purposes as evidence of specific proficiency claims). As discussed further on, this isomorphic property—subject to empirical verification—has enormous benefits for using “data-hungry” MIRT models as part of an operational formative assessment system. The second aspect involves developing a task model map for each scale. Task model maps replace more traditional content blueprints with a highly detail test assembly specification that simultaneously represents the test-form content and the statistical measurement information target(s). Figure 5 depicts the alignment between items, task models, and complexity design layers that control the difficulties (locations) of the task model families. Task complexity is increased (left to right) along each class of complexity design layer metrics. Each task model family is represented by one or more item models that can generate multiple, isomorphic instantiations. The task model map represents the comprehensive content, cognitive, and statistical blueprint for building parallel test forms on a single scale. So how does Assessment Engineering task modeling address the challenges of formative assessment and MIRT There is a to that First, we need to that each scale is an of incrementally more task We need to map the proficiency claims for each scale as a unique construct of all other constructs. Assessment Engineering construct the of domain-specific claims and expected evidence to support those claims. we develop task model maps that provide the of incrementally more task challenges along each domain-specific scale. scales are likely to different of complexity design layers by design. we do not test design by adopting MIRT models that we expect to nuanced within or across task model families. There are three to generate the item families for each task model using item building or item software and it for each domain of or large language models and to produce the items. item can to be when of items task model family are needed (e.g., items The item should use that their in generating the items. quality control can be that language tools and to help the item each family. item is using a software items within each domain are by particular or that can on multiple subject to constraints & & & et al., language and can also be to out some of the generated items. can produce or of items from a Finally, the use of large language models and is still new for operational test There has been some in language testing using large language models based on and most of these applications still on and of the generated test A more in-depth discussion of these technologies and their is the scope of this The point is that we must focus on the intended properties of the task model families. We can then our for the task models to the other formative scales we are to build It is possible to items as isomorphic within task model families and use a calibration where the data are for each family et al., 2010; & the of of the model families item characteristics needs to be empirically verified to an of (e.g., Luecht, 2024; A further would be to each formative trait as a scale for calibration can be such that individual items and families only a single score scale (i.e., the item discrimination has only one nonzero to the domain two to which it There are calibration and scale benefits associated with the item parameters for each domain using a simple That is, under that each item family only has one discrimination to the intended the model uses a single for each family. The difficulties are from that same However, the also a for the item within a family. that of responses is used to estimate each difficulty the model. It should be that there are to the proposed model in the that could be to a MIRT These the IRT models proposed by and and et al. or of test model or trait model It is important to that the model of it to be to with the of rather individual and and et al. Item families are by a design and content that consistent multidimensional It should further be that we may not need models for operational formative We on to the complexity dimensions that should into task and item design. design is of Assessment Engineering and a core of The is that our psychometric models should the intended scale design in as simple and a as or that performance one could apply to task model families of individual items. MIRT modeling has been limited to studies or empirical exploratory factor studies with essentially unidimensional data and score scales. This article makes the that the benefits of MIRT modeling can only be we its capabilities to a test purpose with test design and development that multiple score scales. assessment may be that However, useful formative assessments with some and For example, it is not that of isomorphic items might be needed for each task model family within each domain to support on-demand That is, formative assessments should be on-demand and provide profiles of multiple, sensitive score scales that to good instruction, curricular design, and student learning. the needed multidimensional measurement information is where principled frameworks Assessment Engineering can help in of design, item calibration of item and quality for the of items. By out each construct and the cognitive complexity of task models needed to progress along we also build into the scale design By and then calibrating task model families rather the individual items, we some of the item needed to on-demand testing in a formative assessment Well-designed task model families can further testing and the need to individual items. The fact that we have to with multiple scales and MIRT modeling is not a under Assessment We merely construct each scale from the up and create a psychometric and test development infrastructure that of items and test forms with intentional quality design our efforts to eliminate all but minor of in the statistical characteristics within task model families and help highly robust and useful scales.