The bewitching AI: The Illusion of Communication with Large Language Models

Abstract We investigate the ‘bewitchment’ of understanding interactions between humans and systems based on large language models (LLMs) inspired by Wittgenstein’s later view on language. This framework is particularly apt for analyzing human-LLM interaction as it treats understanding as a public phenomenon manifested in observable communicative practices, rather than as a mental or computational state—an approach especially valuable given LLMs’ inherent opacity. Drawing on this perspective, we show that successful communication requires not merely regularity in language use, but constancy in maintaining reference points through agreement in both definitions and judgments. Crucially, LLMs lack the constancy needed to track negations and contradictions throughout a dialogue, thereby disrupting the reference points necessary for genuine communication. The apparent understanding in human-LLM interactions arises from what we characterize as a ‘bewitchment’: the interaction between LLMs’ statistical adherence to linguistic patterns and humans’ tendency to blindly follow familiar language games. Moreover, when interaction with LLMs is based on stereotyped contexts in which the system seems capable of identifying reference points, we humans automatically apply the practical principle that there is understanding until proven otherwise. The bewitchment becomes thus more profound as LLMs improve in modeling stereotypical aspects of human interaction. This improvement, far from addressing the highlighted limitations, can only deepen the illusion of understanding, raising significant concerns for meaningful control over such systems.