The Impact of Metacognitive AI on Appropriate Reliance in AI-Assisted Decision-Making: The Role of Trust Resilience and Critical Thinking

As AI becomes increasingly integrated into high-stakes decision-making, ensuring appropriate reliance—users’ ability to calibrate their trust based on AI reliability—remains a critical challenge. Metacognitive AI, which monitors and regulates its own decision-making, has the potential to improve trust calibration by fostering trust resilience and critical thinking engagement. However, its enhanced self-reflective capabilities may also lead to over-reliance due to its perceived authority. Drawing on dual-process theory and obedience to authority theory, this study investigates how metacognitive AI influences user reliance behaviors. Using a between-subjects experimental design, 200 participants will interact with AI advisors exhibiting high or low metacognitive ability in a medical diagnosis task. We examine the effects on trust resilience, critical thinking, and reliance patterns, moderated by Need for Cognition (NFC). Findings will contribute to the design of AI systems that foster appropriate reliance and decision-making autonomy, reducing automation bias and improving human-AI collaboration.