What Is the Best Word to Describe AI Uncertainty: Confidence, Certainty, Reliability, or Accuracy?

AI systems are increasingly common in decision-making tasks. However, these systems often generate probabilistic responses, resulting in uncertainty in their outputs. Understanding this uncertainty can help users make robust decisions, such as when to trust the agent and whether to seek more information. Interpreting uncertainty is not always straightforward, though; past work shows there is variability in how people interpret probabilistic information (e.g., weather forecasts). This study builds on that research by analyzing how people interpret probabilistic AI responses. In this survey-based experiment ( N = 632), participants interpreted AI uncertainty statements. The word used to describe the uncertainty (confidence, certainty, reliability, or accuracy) varied between trials. Participants often treated confidence and certainty as indicators of how sure the agent was about its output, whereas interpretations of reliability and accuracy were more variable. These findings can guide how AI systems communicate uncertainty, supporting more accurate user interpretations and robust decision-making.