When the Chatbot Keeps Saying Yes: A Runtime Warning Number for Delusional Spiralling. Which Company Will Do It First in the World?
People are falling into false beliefs after long conversations with AI chatbots. The cause most often named is sycophancy, the chatbot's habit of agreeing with and praising whatever the user says. A recent MIT model shows that even a perfectly rational user can be pulled into a spiral. It also shows that two obvious fixes, a chatbot that says only true things and a user who has been warned, reduce the problem but do not remove it. This concept note points to a gap. Existing studies measure the spiral afterwards from chat logs, and existing guardrail tools check one message at a time. In the searches made for this note, no published method was found that calculates a warning number from the running conversation and lets the computer decide a shunting action from it, without waiting for a person. The note describes who could use such a monitor: makers of chatbots, platforms that supply ready-made chatbots to businesses, and companies that sell guardrail tools. It ends with a question: which company will be the first in the world to do it, and declare it? Keywords: sycophancy, delusional spiralling, AI chatbots, early warning, warning number, shunting action, runtime monitor, AI safety
Authors
- Prakash Vaithyanathan (ORCID: https://orcid.org/0000-0001-8230-4830)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-03
- DOI
- https://doi.org/10.5281/zenodo.23911896
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00