Developing COLLAT: Initial validation of a multidimensional collocation awareness assessment using signal detection and Rasch modeling

Research on multiword expressions—including collocations, idioms, and lexical bundles—has advanced substantially; however, most assessments focus on accuracy- and knowledge-oriented measures rather than the awareness processes underlying that knowledge. This article reports the development and initial validity evidence of the Collocation Awareness Assessment Tool (COLLAT), a theoretically grounded instrument operationalizing collocation awareness across three dimensions: cognitive noticing, metacognitive awareness, and metalinguistic awareness through identification, self-report, and binary forced-choice judgment tasks, respectively, with the identification and judgment tasks accompanied by written justifications. Data from 162 Vietnamese university EFL learners were analyzed within an argument-based validation framework. Signal detection analysis operationalized cognitive noticing as the discrimination of target collocations from non-target word combinations in context, distinguishing perceptual sensitivity ( d′ ) from response criterion ( c ). The one-factor representation of the self-report component was retained as the most parsimonious and interpretable solution. Rasch modeling of the judgment task indicated orderly item difficulty, although targeting favored lower-to-mid ability and precision was limited at the upper range. Mixed-effects modeling supported the cross-model consistency in item difficulty estimates. Analyses of relationships among the three COLLAT dimensions revealed limited alignment between quantitative performance indices and qualitative reasoning depth. Overall, findings provide initial support for interpreting cognitive noticing, metacognitive awareness, and metalinguistic awareness as related but potentially separable dimensions of collocation awareness. COLLAT may therefore complement knowledge-focused collocation assessments by providing diagnostic and research-oriented evidence about how learners notice, evaluate, and explain collocations.