Measuring and Mitigating Stereotype Bias in Language Models: An Overview of Debiasing Techniques

This paper provides an overview of methods for measuring the stereotype bias of pre-trained language models. It explains the term ‘stereotype bias’ and its measurement. A comprehensive review of language model debiasing techniques, including Dropout, Counterfactual Data Augmentation, Iterative Nullspace Projection, Sent-Debias, Self-Debias, Context-Debias, FairFil, and Auto-Debias, is provided. Subsequently, these techniques are compared, and their advantages and disadvantages are highlighted. In addition, there is an explanation of how these techniques work simply. In the future, debiasing techniques will be tested on pre-trained models for the Slovak language, focusing on the task of toxic language detection.