Ontological realism as a validity criterion in second-language strategic competence assessment

Abstract Strategic competence, conceptualized as the ability to put semantic, grammatical, and pragmatic knowledge into use, is a key element in models of communicative language proficiency but remains a difficult construct to assess in language tests. In the oral proficiency interview (OPI), strategic competence is typically assessed through the use of role-plays with a complication. Assessment of test-taker performance on the role-play is subjective and is contingent on raters accurately identifying interactional evidence of strategic competence. Accordingly, validation of the strategic competence exhibited in role-plays has been mostly interpretive. To obtain evidential support for an interpretive argument that role-plays can indeed isolate and provide assessment evidence of strategic competence, the criterion of ontological realism is applied in this study. Towards that end, eleven samples of English-as-a-foreign-language OPI role-plays with a complication were judged by 52 untrained English native speakers. Evidence in support of the ontological validity of assessing strategic competence via role-plays is presented through analyses of the untrained raters’ judgments, augmented by quantitative analyses that identify sources of variation among the raters, including a post-study additional round of coding in which the notion of “success” in the role-plays was examined more granularly than can be done with dichotomous decisions.