An Empirical Bayesian Approach for Testing the Fairness of Automated Scoring

This study proposes an empirical Bayesian approach with loss functions for testing the fairness of an automated scoring algorithm. The proposed method outperforms traditional approaches by (1) being robust to small samples, (2) incorporating parameter uncertainty, and (3) balancing the loss of keeping versus flagging items. The impact of flagging potentially unfair items on the classification of examinees is investigated. The effectiveness of the proposed method is illustrated through simulations and an application to language assessment data.