Fairness and social bias quantification in Large Language Models for sentiment analysis

Large Language Models (LLMs) have enhanced various Natural Language Processing (NLP) tasks, including text generation and classification. However, studies reveal that LLMs exhibit social biases, such as associating certain occupations with specific genders. While previous research has focused on bias in text generation, limited attention has been given to text classification, such as sentiment analysis. This study quantifies social bias in sentiment analysis using five open-source LLMs: BERT, GPT-2, LLaMA-2, Falcon, and MistralAI, fine-tuned on two large social media datasets, the first one is related to nuclear energy, while the second dataset is general and contains tweets related to various subjects. We conducted approximately 1,500 prompt experiments with variations in words reflecting energy source, gender, politics, age, and ethnicity. A fair language model should provide the same sentiment for both prompts; differing sentiments would indicate bias. Explainable methods were employed to analyze how words related to the five subjects (energy, gender, politics, age, ethnicity) contributed to the sentiment. Findings show that social bias persists in LLMs for sentiment analysis, and while fine-tuning can enhance fairness, it does not always eliminate bias, particularly regarding age groups.