Exploring Fairness and Explainability in LLM-Generated Support for Online Learning Discussion Forums

Large language models (LLMs) hold significant potential to enhance online learning by automating responses to learner queries and offering personalized, scalable support. However, concerns about bias in LLM-generated responses present challenges to their ethical and equitable use in educational settings. This study explores fairness and explainability in LLM generated replies within online discussion forums. Specifically, we fine-tuned three state-of-the-art LLMs (GPT-2, Gemma, and LLaMA) using both the original MOOC Posts dataset and a counterfactual version. We then analyzed the sentiment patterns of LLM-generated replies and compared them with human-generated responses. To quantify potential sentiment bias, we introduce absolute distributional sentiment divergence (ADSD) to measure disparities across sensitive attributes, with gender used as a case study. To mitigate bias and enhance transparency, we employed counterfactual fine-tuning by incorporating both factual and counterfactual data, and we used TIGERSCORE, a reference-free explainability metric, to assess response quality. Our findings reveal that LLM-generated responses are generally more neutral than human replies but exhibit varying degrees of sentiment bias across gender. Notably, counterfactual fine-tuning shows promise in reducing this bias, resulting in more balanced sentiment distributions. Additionally, explainability analysis indicates that while newer models (Gemma and LLaMA) outperform GPT-2 in response quality, gaps in accuracy and comprehension remain. This study advances the understanding of bias mitigation and fairness evaluation in LLM-generated educational support, contributing to the development of more equitable, transparent, and responsible AI-driven tools for online learning environments.