Are You Sure About That: Eliciting Natural Language Confidence Indicators from Chatbots

Large language models (LLMs) are becoming the standard for tackling complex natural language processing tasks, but the lack of transparency and explainability of closed-source, publicly available tools that leverage LLMs, such as chatbots, raises concern. Understanding the relationship between the training data, model parameters, and final outputs is particularly important when chatbots are engaged in dialogue from domains where information is continually evolving and the quality of available information can be poor (e.g. health, climate change, and elections). In this work, we investigate the difference between the research and reality of eliciting calibrated confidence scores from chatbots on three types of information: scientific fact, common knowledge, and misinformation. Our objective is to evaluate if a chatbot can accurately label the veracity of various information claims and express their uncertainty in natural language that humans can easily interpret. Focusing on GPT-4 and Claude-3, we find that there are considerable differences in the chatbot responses containing uncertainty and the range in confidence scores elicited. We also find that self-reported confidence scores are not well calibrated and do not directly map to accuracy of response. Our results highlight the need for developing standards for uncertainty quantification in order to help everyday users understand the quality of information they are obtaining, particularly in information gathering interactions where a chatbot can spread misinformation.