Cognitive Biases in Large Language Model based Decision Making: Insights and Mitigation Strategies

Recent development of Artificial Intelligence (AI) has led to an increasing integration of Large Language Models (LLMs) for real-world decision-making. While these models can enhance decision-making processes, they remain susceptible to the same cognitive biases found in humans. Our study examines two well-documented biases found in LLMsAnchoring and Framing effectand evaluates their presence in OpenAIs GPT-3.5 turbo, GPT-4o, and GPT-4o-mini. Using a structured framework, we assess the degree of these biases across different models and proposed four bias mitigation prompting strategies: Chain-of-Thought (CoT), Thoughts of Principles (ToP), ignoring anchor hints, and reflection. We evaluated the bias reduction effectiveness of each prompting strategy using both our proposed prompts and GPT-generated prompts. Our findings reveal significant differences in bias levels across models (p < 0.01), with stronger models exhibiting a greater susceptibility to bias. However, none of the tested prompting strategies indicated statistically significant effectiveness in bias reduction. Our results highlight the challenges of debiasing LLMs and underscore the need for caution when deploying them in real-world decision-making scenarios. Further research is necessary to explore broader cognitive biases and develop more effective mitigation strategies.