Fairness in the Eye of the Beholder: How Gender Bias Alignment Influences Fairness Perceptions and Reliance in Human–AI Teams

Human-AI collaboration is increasingly relevant in consequential areas where AI recommendations support human discretion. However, human-AI teams’ effectiveness, capability, and fairness highly depend on human perceptions of AI. Positive fairness perceptions have been shown to foster trust and acceptance of AI recommendations. Yet, work on confirmation bias highlights that humans selectively adhere to AI recommendations that align with their expectations and beliefs—despite those recommendations not being necessarily correct or fair. This raises the question of whether confirmation bias also transfers to the alignment of gender bias between human and AI decisions. In our study, we examine how gender bias alignment and the presence of explanations accompanying the AI recommendation influence fairness perceptions, human self-confidence, and reliance on the AI system. Findings from two 2 \(\times\) 2 between-subjects studies underscore the relationship between gender-bias alignment, fairness perceptions, and reliance, demonstrating that merely constructing a “formally fair” AI system is insufficient for optimal human–AI collaboration. Ultimately, AI recommendations are likely to be overridden when the system's biases do not align with those of its users.