What Is a Wrong Answer Worth? The Out-of-Sample Value of the Chosen Option in Multiple-Choice Tests

Right/wrong scoring of multiple-choice items discards information about which wrong option was chosen. IRT models that use the choice date from 1972, but evidence on its value has come mostly in sample, one test at a time. We compared option-level and right/wrong models on predicting the option chosen in held-out responses (log-loss, in bits) across 116 Item Response Warehouse tables (110 Brazilian), 469 TIMSS and PIRLS tests and 121 Czech exam forms. In the warehouse, modeling the chosen option gained 15.4 millibits per response over the 2PL, 1.36 times the gain from estimating item discrimination. Of the option's gain, 84% came from which wrong option was chosen, where right/wrong models use only distractor popularity; the rest, on whether the answer was right, was about a fifth of discrimination's gain. The gain was positive in all 116 warehouse tables, in 119 of 121 Czech forms and, with priors on both models, in 463 of 469 TIMSS and PIRLS tests; relative to discrimination, it ranged from 0.8 (Czech exams) to 1.5 (ENEM). By our declared measure, the gain peaked in the lowest ability deciles in most tables; per wrong answer, the choice improved prediction over distractor popularity most in the top decile.