Applying multivariate generalizability theory to compose and rank the evaluation scores of college teachers’ teaching ability

The classical test theory and univariate generalizability theory are difficult to solve how to compose and rank the evaluation scores of college teachers' teaching ability effectively. However, multivariate generalizability theory (MGT) can make up for their deficiencies and is suitable to make decisions for the evaluation of college teachers' teaching ability that shows its new role. 568 students from 16 classes across three colleges in China participated in evaluating 16 teachers' teaching ability using the Teachers' Teaching Level Evaluation Scale for Colleges (TTLES-C), and a multivariate nested unbalanced design (s^·: t^·) was used to collect the data. The mGENOVA software was used for the analysis of MGT. The results showed that: (1) The total dependability was relatively high but some local dependability was low. (2) The original priori weight was not optimal, and changing it could lead to a higher dependability. The optimal weight was 0.22: 0.17: 0.24: 0.37. (3) The MGT could provide the minimum student numbers for evaluation. At least thirty students were needed if a 0.80 multivariate index of dependability was desired. (4) Among the 16 teachers, the highest composite universe score was the 13th teacher (25.05011) and the lowest was the 15th teacher (23.58889), but the composite estimated conditional absolute errors of the 12th and 15th teachers were relatively large and their ranking scores lack accordance. Based on considering optimal weight and minimum student numbers, the MGT has a new role to effectively compose and rank the evaluation scores of college teachers' teaching ability, which is worth to recommending in actual evaluations.