Measurement Reflections

I am very pleased to be one of the former NCME Presidents to contribute to this special issue on The Past, Present, and Future of Educational Measurement, as serving in that role was a career highlight for me. In my article, I review events in my life in measurement and how there may be lessons to be learned by others from them. This review gives the article a different flavor than most of the other contributions, but I hope readers will see relevance in my experiences to some of the decisions and situations they now face or will encounter in the future I have been employed in standardized testing for 60 years, the majority of which were spent at large testing enterprises. Much of the time I oversaw test and test program development including admissions tests, state and school assessments, and tests used in counseling, employment, and clinical settings. I estimate that many tens of millions of students have taken tests developed by my staff. I have been very active in the testing industry and profession over my career, serving on boards and committees, editing this journal, and filling the role of President for the NCME and the Association for Assessment in Counseling on the professional side and the Association of Test Publishers, the core test industry group. In my reflections, I am trying to look back on events that stand out for me and their importance at various stages of my career. Perhaps readers can draw some lessons of value to them. I tell my grandchildren that they will learn a lot from their own experiences, but that they should try also to gain from observing and thinking about the lives of others. If you keep your eyes and mind open, there is so much to learn. I finished college at night while working full time so did not have many incidental contacts with college services such as counseling. I had one required appointment with a counselor. He asked about my plans, and I replied that I wanted to continue with some aspects of psychology as I had had exceptionally good experiences in my coursework. When I said I liked every part of psychology, the counselor told me that educational psychology would be a good direction for me. I left my one and only college-level counseling session motivated to explore graduate study in educational psychology and that search led me to Teacher's College, Columbia University. This path led me into testing as a career. I am comfortable with the outcome of my decision, but I believe that more attention to career counseling for part-time students would be highly desirable. After completing college, I was sitting at my desk in the accounting department where I was working when I received a call from a Professor at Teacher's College where I had applied for graduate admissions in Educational Psychology and Measurement. The professor was Robert L. Thorndike and he wanted to know if I would be interested in receiving an NDEA Fellowship and additional university benefits as a Teacher's College PhD student? I was thrilled and accepted immediately, a life-changing decision that opened so many doors for me. The faculty included leaders in the field of measurement and my fellow students were also all people who wanted to make a difference in the world. We all saw testing as extremely valuable and wanted to play a part in developing and using tests. I kept track of some of my fellow students and they became productive contributors to universities and to the testing profession and industry. Fellowships and other support help open doors for many students, particularly when they are the first in their family seeking college and graduate degrees. Robert L. Thorndike used an example in a graduate school class presentation of an unintended consequence of the strategies measurement experts developed and used for selecting aircrew in WWII. The highest math scorers in a group of very able men were trained as Navigators rather than Pilots. After the war Pilots were more likely to stay in the service and rise in rank so the senior Air Force Leaders in future years were not the strongest math folk available. Not an intended outcome or one even considered at the time of test use. Thorndike urged us to always try to foresee likely consequences of our plans for testing and to follow up on how tests were actually used. What impacts were occurring for the diverse groups that were involved? I think about this issue whenever I am evaluating the way a test is being used. We all need to do this in all phases of testing. There is a tendency to accept some features of testing as inevitable once we have produced a strategy for dealing with what we recognize to be a problem. Realizing that unequal familiarity with what a test will cover creates an unfair situation, we strive to provide practice examinations to all test takers and to create comprehensive preparatory materials accessible to all. We should not settle, though, for the status quo. We should be seeking to make our exams and how they are used as helpful as they can be to achieve productive social and educational goals. That requires us to listen to those who criticize our work and to find ways to involve those impacted by testing in the development and evolution of tests as well as testing policies and procedures. The former President of ETS, Henry Chauncey, was speaking to a group of us who were participants in the 1964 ETS Summer Program for Graduate Students in Measurement. He said that people felt that a measure of motivation would be a good addition to the SAT when selecting students for college. In somewhat of a tongue-in-cheek way, he offered the idea of an electrified grid that students who were barefooted would need to walk across to get into a particular college. This notion was not meant to be practical and would fail many important DEI standards, but it did raise interesting issues about the measures we did use. What did they leave out? Who do they favor? I see many exchanges in my Email flow proposing innovative approaches to the use of tests in educational transitions. One common element in the discussions is how testing can be combined with instructional activities. If we could obtain the information to help students move up educational levels from how they handled prior learning, we could broaden the group about whom we had relevant knowledge and not rely as much on single- occasion tests. This would help decrease the continuing impact of Testwiseness on performance, especially on multiple-choice tests. When I was touring the ETS Campus as part of the Summer Program, we visited a section that responded to requests for student's SAT scores. I was puzzled about the fact that students were making these requests. I had not realized that this was possible as SAT scores were not sent to test takers at the time I took them as a high school student. Checking in the literature I found that up until the late 1950s, The College Board and ETS judged that students should not receive their scores as they might place too much weight on the meaning of the scores. That concern had merit, but the policy changed and now along with all other students I could see my own score. We are still addressing how to present scores on tests to test takers. The move over my career in testing is to provide increasingly greater amounts of information. One of the challenges is to try to communicate the strengths and limitations of standardized test scores. As with any sample, test scores tell only part of the story about the test taker. How can we best make that part meaningful and productive? Are we doing all we can not only to encourage proper use of test scores, but to guard against overuse of test scores including applications of scores for which no supporting evidence has been provided? As we evaluate the validity of uses of tests used in admissions at any level, we need to keep trying to communicate in a way that test takers and test score users can understand clearly. It is not enough to develop high quality tests; we want to facilitate high quality uses of scores. That must include recognizing misinterpretations and correcting them as best we can The distinguished ETS psychometrician, Fred Lord, once and only once came to my ETS office for advice! What was this Criterion Referenced Testing idea about which he was reading? He recognized how uninformed about measurement principles most of the CRT writing was. I was able to point out a core of meaning that made sense and to identify ways that CRT had the potential to be a meaningful approach. It was a brief meeting, but a career learning experience. The top measurement scholar was staying on top of new developments and turning to colleagues, including ones quite junior to him with questions. As we become more expert, we need to continue to listen to those at earlier stages of their career. The newer voices are not as likely to channel their thoughts down the same paths even as tools and challenges evolve. I once read that a senior business executive had instructed his support staff to allow access to anyone with “completely crazy ideas.” I cannot say that I followed this rule myself, but I have tried to keep in mind the real risks of only listening to ideas solidly in your comfort zone. When you are making changes in testing programs, I urge you not to say that the change is going to be “easy.” This triggers responses such as we give when someone says, “This isn't going to hurt.” In many instances test score users are proceeding as they were taught years ago. They need to learn a new set of interpretations and explanations. The relationship between the number of questions answered correctly and test score may now be harder to understand and to communicate about. You need to explain why you are making changes and to show how you will help. The better you are at providing a productive context, the more support you will earn. Just do not say it will be easy. In 2003, at the annual Association of Test Publisher's (ATP) Conference, I met with David Foster, someone whose judgment and working style I had very much respected in roles we had at ATP. I was thinking about a career move and had five possible directions to get his help thinking about. The third on my list was test security. I had been helping design and deliver workshops for an ETS teaching/consulting program—The ETS Global Institute. Regardless of the country the workshop participants were from, cheating on tests and related security issues came up as a major concern. There seemed to me to be a place for test security services as a line of business. I never got past that item on my list. Dave said that he and a few Salt Lake City area colleagues had reached the same conclusion and were going to “meet at his cabin” later that year. Dave invited me to join them. There was no outside investor or fund to cover the airfare for me or other non-local folk coming to the meeting, but we could all stay at Dave's cabin. The “cabin” proved to have 6,000 square feet of space in a lovely home in a forested area. A dozen of us contributed to work already underway to define the company and the initial services of Caveon Test Security. I sometimes tell new measurement professionals that our professional conferences are exceptionally effective ways to build a network. There may be a conversation like the one I had with Dave Foster in their future. If you are helping professional staff develop, encourage them to attend conferences related to their work. Provide funding where you can do so. Make sure that your staff know that you are expecting them to stay abreast of developments in the measurement profession and industry. That is part of their job, not a nice-to-do digression. At conferences, participants should devote part of their time to sessions related directly to their work. When a session of this type ends, participants should go to the front of the meeting room when there are presentations meaningful to them and thank the presenters. When really impressed by a speaker, the attendee should try to establish a relationship with that speaker. One of the consequences of the growth of electronic and virtual meetings is that you need to take special steps to connect with interesting people who you have heard. There is no “front of the room” to go up to. Take steps such as sending an email, even if this does not “come naturally” to you. At in-person conferences, view sessions, receptions, refreshment breaks, and exhibit halls, as a chance to ask questions and always listen carefully to answers. It is a wonderful way to learn and grow as a measurement professional. Looking back at the different jobs and roles that I have played in testing; I am struck by a continuing need to help people make decisions. Whether it is in an educational, employment, or certification setting, someone is deciding about the qualifications or appropriateness of groups of people. Should this person be hired or admitted? Which of these people has shown that they merit certification of a set of specialized skills and knowledge? I have learned that the more I know about the circumstances for the decisions the more helpful I can be. What are the consequences for the test taker, the test program sponsor, and society of an inaccurate decision? Some misjudgments based on tests do not cause much difficulty, as when a test taker is asked to proceed more slowly through the next set of lessons. This outcome may slow a student's progress but might not have other worrisome consequences. In sharp contrast, in high-stakes testing settings, careers may be sidetracked, or harm caused to the public due to an incorrect decision about competence based on standardized tests. For our work in educational and related testing, we need to adhere to a rule from carpentry “measure twice, cut once.” We need to devote time to learning from potential test users and those who will take our tests what the impacts will be. Then we must continue to take the measure of our own work going forward. That is our responsibility as measurement professionals. In the foregoing paragraphs, I have shared some of my career experiences and the lessons I draw from them. If you have reactions that you would like to share about anything I wrote, I would love to hear from you. We need to keep learning from each other for testing to continue to evolve as the world changes about us.