The Efficient Development of Conflict Structure Datasets for Evaluating Sentiment Recognition Bias in Large Language Models

The rapid advancement of AI technology has led to the development of large language models (LLMs) that acquire extensive general knowledge from vast amounts of text data and are utilized in various tasks. However, LLMs also inherit biases from their training data, resulting in discriminatory behavior related to gender, race, and political ideologies. In the field of national security, sentiment recognition bias towards specific countries poses significant problems. Previous studies have developed datasets to evaluate these biases, but several issues remain in their methods. This study proposes a new method for developing datasets to evaluate sentiment recognition bias in LLMs, using tweet data related to the Ukraine-Russia war. The method involves automated sentiment labeling and anonymization using LLMs to create efficient and high-precision datasets. GPT -3.5-turbo was used for initial sentiment labeling, with human corrections applied to select tweets positive towards Ukraine and negative towards Russia. Anonymization processing with LLMs ensured specific names did not influence bias evaluation. Experimental results confirmed the proposed method effectively evaluates sentiment recognition bias in LLMs across various conflict structures. Metrics such as Emotion Inversion Consistency Rate (EICR), Positive Odds, and Negative Odds quantitatively assessed biases. The results clearly demonstrated LLMs‘ emotional biases towards specific countries and their recognition of conflict structures. In conclusion, this study presents a new method for evaluating biases in LLMs and demonstrates its effectiveness. Future research will focus on expanding dataset size and improving anonymization to further enhance fairness and reliability. This study lays the groundwork for more robust evaluations of LLM biases.‘’