Subscores: A Practical Guide to Their Production and Consumption. ShelbyHaberman, SandipSinharay, RichardFeinberg, and HowardWainer. Cambridge, Cambridge University Press2024, 176 pp. (paperback)

Educational and psychological tests often have different subsections based on content categories or blueprints. For example, a test on mathematics knowledge may have subsections on arithmetic, algebra, and geometry. Scores assigned to these subsections are commonly known as subscores. Interest in subscores surged during the first decade of the twenty-first century possibly due to their potential remedial and instructional benefits. According to the National Research Council report “Knowing What Students Know” (2001), the target of assessment is to provide particular information about an examinee's knowledge, skills, and abilities. Subscores might have the potential to provide such information. Similarly, the U.S. Government's No Child Left Behind (NCLB) Act of 2001 demands, among other things, that students should receive diagnostic reports that allow teachers to address their specific academic needs; subscores could be used in such a diagnostic report. Naturally, there was and still is substantial pressure on testing programs to report subscores, both at the individual and aggregate levels such as at the level of institutions or states. So, it comes as no surprise that subscores are reported by several large-scale testing programs, such as SATR, ACTR, Praxis, and LSAT. Despite the demand for subscores and their potential remedial and instructional benefits, subscores have their limitations as these subscores are usually based on very few items (i.e., less reliable) and are often highly correlated with the already reported total scores, thus making them psychometrically unsound to report. This recognition of the inadequate psychometric properties of subscores led many researchers and practitioners in the field of educational measurement to start researching subscores and publishing their findings in papers that cautioned users about these potential problems with subscores. Many of these papers also showed procedures developed by researchers that may allow testing programs to report subscores that are more precise and informative (e.g., Haberman, 2008; Sinharay et al., 2007; Sinharay, 2010; Sinharay, 2014; Wainer et al., 2001). Despite the slew of prolific work done in the area of subscores, there was a lack of a single and coherent source summarizing this important work that testing practitioners and score users can refer to when working with subscores. Subscores: A Practical Guide to Their Production and Consumption by Haberman et al. (2024) is a worthwhile attempt to fill this vacuum by providing a single and coherent source summarizing the very important work done on subscores. The book comprises six chapters. Chapter 1 provides an account of the history of scores and subscores. Chapter 2 focuses on the reporting of subscores and provides several examples of how subscores are reported for many tests. Chapter 3 focuses on subscore quality and psychometric ways to assess it. Chapter 4 provides survey results about the quality of subscores for many operational tests. Chapter 5 summarizes some strategies to follow when subscores are in demand but do not satisfy adequate psychometric standards. Finally, chapter 6 includes some concluding remarks with some of their recommendations for practice. This book review will follow the same separation of chapters and also provide some concluding remarks. Chapter 1 provides a comprehensive summary of the history of scores and subscores and forces the reader to think about why tests need to be as long as they typically are. Figure 1.1 shows how the law of diminishing return sets in quickly after the test has a certain number of questions (e.g., 30). So why do tests have more than 30 questions for example, unless the small additional increment in reliability is of any practical importance? The U.S. consensus example attempts to clarify this by showing that estimating the total population size (compare this to the total score on a test) for a given year is relatively easy and cheap but not so for very small geographic regions (compare this to subscores on a test) which requires massive data collection and therefore very large expenses. This example conveys two important messages. First, obtaining an accurate estimate for the grand total (or total score) is easy and cheap. Second, obtaining accurate estimates for smaller units within the total (or subscores) is hard and expensive and should not be attempted unless there are serious practical reasons for doing so. I have two main comments about the example. First, the Constitution and acts of Congress require accurate details on small groups for reasons such as the appointment of the House of Representatives, allocation of social services, etc. That is why they spend so much on the census. The total population of the United States comes out as a byproduct of accurate estimation of small areas. In testing the primary goal often is (unless otherwise stated) to provide a total score based on a reliable and valid test that can be used for purposes such as admission or certification. Subscores usually come as an afterthought. Although I see the point the authors are trying to make, it seems the consensus example is in reverse order of how the testing world works. A second minor comment is that it seemed unnecessary to utilize more than two pages of the book with the consensus example when the authors could have simply mentioned that if subscores were deemed important, then more items for each subsection are necessary to produce subscores that are reliable and valid. Furthermore, this example is also found elsewhere (Wainer, 2015) and could have simply been referenced. The next section in this chapter goes back to the testing world and highlights the point that the production of meaningful subscores needed for important purposes (e.g., admission, licensure, and remediation) would be a justification for tests that contain more items than would be required merely for an accurate enough estimate of the total score. The sections on the long history of scores and subscores and the origins of mental testing in the U.S. military are informative but in places were probably unnecessary and strikingly bold with sentences such as “the banks of the Jordan were strewn with bodies of the 42,000 who failed” and “the testing process was so rigorous that sometimes candidates died during the course of the exams.” Also, the long section about the ASVAB although quite informative could have been shortened considerably. At times, it felt like a manual for the ASAV was stuck to this chapter of the book. Having said that, this chapter provides a very good account of the history of scores and subscores. The summary section is a very good “go to place” when designing or redesigning a test with certain goals in mind. It will encourage psychometricians, test developers, policymakers, and test users to ask questions about important factors such as whether the total score relevant for its potential use, is the total score reliable, are the subscores reliable, and do the subscores provide information that is not already provided by the total score. Chapter 2 provides a summary of the various ways subscores are reported for many large-scale testing programs. This chapter starts with a story about Daisy, a fictional character based on a true story. In this story, Daisy's parents demanded that her school grant her curriculum and testing accommodations based on a single math subscore which made it appear as if she was weak in math although in school she always performed above average in math. The story highlighted the complexity of subscore reporting and how it can be misinterpreted and misused. The next section showed (through 4 different figures) how for a hypothetical student and test composed of five subcontent areas, subscores can be presented and interpreted in several different ways (e.g., presenting raw subscores, percent correct and percentile scores, etc.). The authors then summarized how subscore information can be displayed for formative (low stakes) and summative (high stakes) assessments. In the next section, the authors discussed different types of subscore users such as test takers, governing boards or agencies, faculty, and administrators and how many assessments report subscores for different purposes and communicate them to these different types of users, which are intended to factor into how a testing program determines what to report. The next section provided a detailed account of how subscores are reported for large-scale tests such as Praxis, SAT, ACT, American Board of Internal Medicine Maintenance of Certification, Armed Services Vocational Aptitude Battery (ASVAB), Duolingo English Test, United States Medical Licensing Examination (USMLE), the Personality Assessment Inventory (PAI). In the Praxis example, a table containing the raw score points earned out of the maximum possible points and the average performance range was provided. Similarly, the SAT sample score report example included an individual's total score and performance on different sections and subscores. For all the examples, some guidance on how to interpret score reports were also provided. The penultimate section provided some guidelines on how to generalize score reporting best practices to subscore reporting best practices. The authors of this book, as well as several authors working in the area of score reporting, recommended a systematic approach to score report development, which includes gathering information about user needs and user characteristics pertinent to reporting, engaging with stakeholders to collect feedback, creating a preliminary report and then creating a final report template that can be used for operational development. Other systematic approaches to score report development are also discussed. In the final section, the authors concluded that the decision on how subscores should be reported depends on many factors such as prospective uses of the subscores, their psychometric properties, the types of tests, and the stakeholders to whom the results would be reported. Chapter 3 focuses on the importance of evaluating the quality of subscores before deciding whether or not to report subscores. For evaluating the quality of subscores, the authors focus on three issues, subscore reliability and validity, the correlation between the subscore to the total score and to the other subscores, and the score reporting itself. The chapter presents these issues in a highly technical format which may appeal to a narrow and fairly technically astute audience. But the key elements of this chapter are presented in the first five pages which summarizes the technical details that followed more substantively and would likely have more universal appeal. The concept of reliability as provided in this chapter is the correlation of the test score (i.e., subscore or total score) and a parallel form test score randomly selected from a hypothetical infinite number of parallel forms. Even when the reliability of subscores is sufficiently high, the authors caution testing programs to not report subscores if the subscore predicts the true subscore less well than does the total score which is assessed using a statistic called the proportional reduction in mean squared error (PRMSE). They also show that the PRMSE for an augmented score that predicts the true subscore by both the subscore and the total score is larger than the PRMSE of the subscore and the total scores separately. In the next section, the authors discuss how in practice score users may want to compare performance in different subscontent areas (i.e., scores on math versus speaking) rather than predicting individual subscores. Their solution to this issue is comparing the true subscore to the true total score using residuals where a negative residual true speaking subscore, for example, would suggest less proficiency in speaking than what is indicated by the score on the total test. The section that followed dealt with issues such as determining how long the test should be which the authors answer using the concept of the value-added ratio (VAR) where a value exceeding 1 suggests that the number of items contributing to the subscores is sufficiently large. They also discuss ways to combine subscores using principal component analysis and provide an example that is quite intuitive to understand. They show that the average proportional reduction in mean squared error (MSE) from the use of the first principal component (with an eigenvalue of 3.44) is only slightly than the by the true total score they used the first two principal as the average proportional reduction in the only but the main point seemed to be that the subscores all a single They also discussed using all subscores to a true subscore but as can be by the results of the principal component the between the principal component analysis and the analysis was a single The of sections with and aggregate level validity, they to answer questions such the speaking subscore provide a the quality of work by a the provided by the total and to the that is by the reliability of the In other if the reliability of the speaking subscore is then the also be aggregate level subscores, the authors subscores at an aggregate level such as students from specific or geographic They by that subscores that are not for might be for a and subscores that are for might not be for The authors this section by making several important recommendations on how to whether or not to report subscores and these recommendations are based on such as reliability of subscores, of subscores, knowledge of test users, and that although this chapter from book chapter by this was not the authors intended to the on subscores into a and source that provides an source for users the issues and best practices the reporting of subscores. there is still content that the authors have such as the about residual subscores, tests and tests, scores, etc. of the (e.g., Figure to the Figure to reliability and of the are also Chapter 4 provides a summary of the subscore analysis results based on data from operational tests. were done in an attempt to answer how well do in Chapter 3 (i.e., if subscores have value to be are for operational tests. The second that these attempted to answer was that for subscores that do satisfy what are the characteristics of subscores that have on and the authors provided of a number of subscores in each test that value and concluded that subscores have value in practice. Although it seemed the was a on the of for tests several subscores from to to have The of the tests subscores that So I that the was true but as a testing I felt the need to not the tests where subscores some on the results from the tests, the authors made very important and practical about the required and subscore reliability to have a of these subscores value and may be were for and for subscore should not be in and may be different on the of the test and the in reporting the subscores. But it provides a very good point when designing or redesigning tests and making about whether to report subscores or not for a particular test. Having the of and subscore the authors then provide information on in Figure and concluded that for subscores to have any of the correlation between two subscores should be less than the reliability of subscores and the are important as the authors point out that for tests that do not have any value-added subscores, it is these subscores lack reliability or or Finally, in the section, the authors provided some caution in these results and these any to test test etc. For example, they mentioned that the tests only or So results from tests with long items may be to test content is also For example, as the authors it seemed very for tests to have value-added subscores but for other tests where the content by subscores was by and more of subscores to have practitioners often have to the that scores on often do not satisfy of psychometric quality in of reliability and and therefore should in good be provided. But with the demand for subscores from users such as test and academic institutions such as and testing practitioners in a be reporting subscores that do not satisfy of psychometric quality is what some testing programs often have to or should for other psychometrically In Chapter the authors attempt to address this very important practical They do so with a recommended that is in an that a testing may when deciding whether and how to report subscores any such as for remedial and instructional The several and that the that the testing might want to For example, the first that is is whether the subscores have This is followed by the and and if the answer is then report subscores and if the answer is then if augmented subscores have This a course of is This in has the potential to of the a testing can have when to on whether or not to report subscores. The authors provide more details on some of their recommended (e.g., reporting augmented subscores) by using a data from a test content knowledge in which subsections content knowledge in and social on their PRMSE they that only the mathematics subscores They also an augmented mathematics subscore for an as a of the examinee's mathematics subscores as well as the examinee's score on the three subsections with the subsections the correlation with the mathematics subsection larger in the They also provided an of an augmented subscore as a of the subscore and the total score. on their analysis of the test data they provided as an example Figure the subscore, the augmented subscores, and the average subscore for five which showed that the augmented subscores to be less out than the subscores. In the of the chapter the authors provide details on other for reporting subscores such as based on the performance or no reporting of subscores, level feedback, reporting of or subscores, the test diagnostic scores based on a diagnostic etc. Finally, in the and section, the authors provide an to some of the discussed in the which will be very for users who want to the and some of the done by the They also that it is to not report subscores when they do not provide any value or report augmented subscores which is fairly to that the has in in other such as and Wainer this chapter has content such as the and the recommended to follow when deciding whether or not to report subscores or augmented subscores with reporting of of test subscores, etc. The section of Chapter 6 as an important that out meaningful and subscore information that is simply not there in a test. if subscores were to be reported to and academic for example, then the for such subscores should be into the test during the of a test. In this section, the authors that if there is a to report subscores for a test and test want to the for subscores during the of the it might be due to the in test and of such a test. In the next section, of the first five chapters are provided followed by a section in which some recommendations for practitioners are provided a very by more detailed on the of the showing how a test can be to provide for the subscores are found to be of inadequate score is as a to But if does not the reliability enough but a testing program has to report subscores due to or then enough caution the subscores should be provided so as not to use them for any serious or purposes (e.g., In the section on score reports that the authors show different on how to the scores on a score report. they showed was in the form of a hypothetical report when the is by the data it (e.g., the from areas where the is to the areas where the is They also that a score report the error of a score so that it the user some information on how much a particular is to due to In the next section, the authors discuss some about subscores. For example, they discuss a commonly that when subscores no value for the total of there might be some specific within which there is Although this is in the authors that shows that value is Haberman, Other discussed in this chapter such as subscores mean weak information than can weak subscores value with etc. But in each of the authors that these are and In the of the the authors provide a practical for the subscores have no value and the reliability of the total test is than This has some such as the of test development, the required to the the to provide subscores that have no etc. I think this the same point made in Chapter 1 Figure should come with a Although it like there is very in reliability after the test a certain (e.g., 30 the of a test for example, to a or test may not be from a content reliability and are important and be before making such a decision to the test. Finally, the authors with they already in the section of this which is to tests with their use and in and then to data to if the have been was a in subscore in the of the twenty-first of this was by an in demand for subscore information at both the individual and aggregate levels to focus on areas that need testing practitioners were to provide this information a subsection with items than the total test may not be to a Furthermore, subscores are often to the total scores and do not provide much value given that the total score which is already reported. issues were by the authors of this book and by other researchers as The authors intended to the on subscores into a and source that provides an source for users (e.g., psychometricians, test developers, and about the issues and best practices the reporting of subscores. such a source not the authors on the of such a source by this book. For users who may not be with the issues and best practices subscore reporting and therefore may not have to the work done on this this book is a source to an summary of the work done subscores. This book can them answer questions such as how many large-scale operational tests report subscores, how to the psychometric quality of subscores to on whether to report subscores or and what are (e.g., score when scores do not psychometric standards. The from Chapter 5 showing a recommended to follow when subscores do not psychometric and should not be reported will be for users working on testing programs where subscores are in Similarly, the in Chapter 6 showing how a test can be to provide for will be when tests or redesigning tests and when it is known that due to or other subscores have to be provided. the book what it to provide a summary of about subscores in a single source which will the for more and on this that I was discussed in a more is the issue of of subscores different of the same test (i.e., subscore if the subscores are used for purposes such as admission or For example, why is it important to subscores and what are recommended ways to subscores using items the test subscores using and total scores as an the test the subscore to the total score which is already on for each form the single subscore more precise when based than score are used for score reporting such as in of testing or should augmented scores be I this review with the that in subscores (or and on other issues subscores for many more I the authors for this book on relevant issues and practices on the of subscores. is an primary test score and use of subscores in large