Deep Learning Approach for Evaluating Fairness in LLMs

Large Language Models (LLMs) are prone to bias that can be introduced in the dataset itself or in the training phase. This can make their output lacking in fairness and hinder certain discriminated groups on the basis of stereotypes that the LLM may have been fed with or learnt. In today’s day and age, LLMs are used to make major decisions and thus it is crucial to identify and remove such bias. In this paper, we have focused on identifying the bias present in medium-sized LLMs such as BERT, RoBERTA, DeBERTa and GPT-1 using bias evaluation metrics such as intrinsic and extrinsic. Both the metrics have been used and bias has been identified in the datasets that the LLMs are being trained on (intrinsic bias) as well as in the outputs of the LLMs for certain queries (extrinsic bias).