When AI Agents Talk, What Information Gets Lost?
When many AI systems work on the same problem, it is intuitive to think that letting them discuss their answers will naturally make the final result better. But this discussion can also cause useful details to disappear as the systems converge on one shared response. I tested this possibility in long-form policy analysis, where it is necessary for a good answer to keep track of many facts and trade-offs. I built CA-TradeoffEval, an evaluation testbed based on official California ballot-measure materials, and broke ten policy measures into 120 small reference claims and 367 smaller information components. I compared four ways of producing a final answer: one AI agent working alone, three independent agents whose work was combined, ordinary multi-agent debate, and debate in which one participant acted as a critic whose job was to catch missing or weakened information by referring back to the source. Across 160 final outputs from GPT-5.6 Sol, critic-guided debate preserved more source-grounded information than ordinary debate by 3.13 percentage points (95% measure-level bootstrap CI [0.53, 6.34]). Independent aggregation had a smaller positive estimate over a single agent, and ordinary debate was almost indistinguishable from independent aggregation. I then ran a smaller replication using Claude Sonnet 5 on five randomly selected measures. Claude had a similar critic effect (+2.62 points), but the confidence interval was wider and included zero. The Claude replication also showed more distortion in critic debate, suggesting that there is a possible trade-off between keeping information but also possibly introducing more inaccuracies. Overall, the results suggest that how AI agents are organized matters, making them discuss is not enough, and a clearly defined critic role can help preserve information, but it should also be paired with checks for distortion.
Authors
- Spriha Pradeep (ORCID: https://orcid.org/0009-0009-9451-7660)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-11
- DOI
- https://doi.org/10.5281/zenodo.22715133
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- preprint