Concept map analysis of reading comprehension assessment: Tracing situation models in multiple-choice and constructed-response items

Reading comprehension assessments are often criticized for focusing on discrete text segments rather than on deeper comprehension processes such as situation-model construction. This exploratory study investigated whether concept map analysis can be used to evaluate how well comprehension items reflect the conceptual structure of an expert situation model underlying a text. The study was not intended as a generalizable empirical evaluation of existing items, but as a methodological demonstration of an analytic approach. Accordingly, 18 selected-response items from Dutch vocational education reading exams were analyzed using expert-constructed concept maps as proxies for situation models. For each item, the number, implicitness, and centrality of required concepts were examined to estimate the extent to which items engage the underlying conceptual structure. In addition, three constructed-response versions of selected items were developed and compared with their selected-response counterparts. Results showed that most selected-response items addressed only small portions of the concept maps, typically less than 10% of the mapped concepts, although summarization items showed somewhat broader coverage. The constructed-response versions engaged substantially larger portions of the maps than the selected-response items. Exploratory correlational analyses further indicated that items requiring more concepts, particularly more central concepts, tended to be more difficult. These findings suggest that concept map analysis provides a useful method for examining construct representation in reading comprehension assessment and for informing the design of items that better capture deeper comprehension processes.