The arrangement of response options in multiple-choice test items: verticality is not always better

Multiple-choice tests are widely used to measure learning outcomes. Consequently, constructing high-quality test items is critical, and many authors have advanced item-writing guidelines. One frequently mentioned guideline is to arrange the response options vertically. However, evidence to support this recommendation is scarce and has only been obtained for items with text-based options. This study aimed at understanding whether the arrangement of options affects performance at solving items with large-sized options, such as graphs and pictures, using objective and subjective measures. Fifty-seven high-school students completed a multiple-choice science and mathematics test with 24 four-choice items, options being graphs or pictures presented in one of four arrangements: vertical without page break, vertical with page break, Z, and inverted N. Response accuracy, response time, and perceived difficulty were obtained for each item. Subsequently, students participated in a cognitive interview about their experiences, practices, perceptions, and beliefs regarding the arrangement of options. Objective measures show that the arrangement of options hardly affected performance, the only effect being that vertical condition with page break resulted in significantly longer response times. Subjective measures show that most students favored the vertical arrangement they consider more common but negatively perceived vertical condition with page break and considered squared arrangements (Z, inverted N) to facilitate visual exploration and comparison between options, as opposed to the vertical arrangement. Results suggest that the vertical arrangement does not offer clear advantages over squared arrangements for items with large-sized options.