Developing a rating scale to assess phraseological competence

The study set out to design, develop, and validate a rating scale for assessing second language test-takers' phraseological competence. It also aimed to examine the relationship between human ratings of phraseological competence and assessed levels of proficiency in the Test of English for Educational Purposes (TEEP). The dataset comprises spoken performances of 127 TEEP candidates placed across six levels of proficiency. Four dimensions of accuracy/appropriacy, diversity, sophistication, and abundance were identified, and a scale of 1-6 was developed. Training was provided to three expert raters and two standardisation meetings were held before they were asked to rate the spoken performances independently. Many-facet Rasch Measurement was used to analyse three facets (i.e., candidates, raters, and criteria) and compute logit scores as indicators of test-takers' phraseological competence. The analysis showed that a 5-level scale worked more effectively, demonstrating adequate overall data fit to the Rasch model. The results suggested that both raters and criteria in the modified 5-level scale worked well and sufficiently contributed to the relatively precise and adequate measurement of phraseological competence for research and proficiency assessment purposes. The ANOVA results revealed a significant main effect of proficiency level on the ratings, implying that phraseological competence, represented by the logit scores, differed significantly across assessed level of proficiency. Areas for improvement in the measurement and implications for language testing are discussed.