Never Miss the Narrows: What a Short Instruction Does and Doesn't Fix When AI Assistants Plan Trips Over Long Conversations
AI assistants now help people plan multi-day trips over long conversations. They can say something true and important once, early on, and then fail to raise it again when the traveler actually decides. They can also quietly trade away a must-see to fit something else, or forget a refund deadline they noted themselves. We call the goal of avoiding these failures informed-choice preservation. We propose a model-agnostic instruction block, the "Never Miss the Narrows" protocol: six scored rules plus one hygiene rule. We test it against a plain baseline and a variant with explicit memory. What we tested: 7 models (4 frontier, 3 open-weight) on 36 scenarios, preregistered and independently reviewed before data collection: long conversations of 120–248 messages at real places, with scripted (frozen) middles and live replies at six decision points each; short cases derived from documented real incidents; constructed cases; and no-nag controls. What the protocol fixes. The most meaningful measure is what a traveler understands after reading a reply, so we asked AI "travelers" to read each reply cold and say what they could now choose and what each choice would gain or cost. Points understood rose from 0.42 to 0.59 (+0.17, 95% CI 0.12–0.23), in all six long cases. Two gains came through clearly. Deadlines the assistant had recorded itself: 58% → 71% of readers knew the deadline and why the booking might not fit; judges and readers agreed almost perfectly on this. Spare time: readers who realised the day would end early went from 6% to 38%. The preregistered judge composite shows a larger gain, 0.49 → 0.76 (+0.27, 95% CI 0.19–0.35). Part of that gap is rubric echo: some judge metrics mirror the protocol's own wording. What it doesn't fix. Models mentioned the must-see far more often, but only 30% of readers could say why it mattered even after protocol replies. Mentioning is not informing. The explicit-memory design we tested, a model-written JSON summary, added little (+0.03 to +0.04) while using about three times the compute. Possible costs. In controls where nothing was critical, the judges rated substantial over-intervention up from 10% to 26% (35% with memory). However, AI readers barely more often wanted to reopen a settled decision (+0.03 to +0.04, not significant), so the size of this cost is uncertain. Decision-critical facts the protocol does not mention were disclosed slightly less often. Replies grew 11–14% longer. Replay of a real failure. Re-running the conversation that motivated this work, current frontier models no longer repeated the original mistake (cutting a walk short before its signature endpoint). The gap that remains is narrower: why the endpoint matters, the self-recorded cancellation deadline, and idle time. Without the protocol, no model raised the deadline. Caveat. All labels come from AI models (three judges, two readers, one grader); human annotation is in progress. We compared the protocol only with a plain baseline, not with simpler disclosure prompts from prior work. The six long cases are scripted around the moments the protocol targets. Working paper v4.3. The attached package (ICP-NeverMissTheNarrows-v0.5.3-public.zip) contains the preregistered evaluation: 36 scenarios, 755 runs across 7 models, judge labels, analysis code, a blinded human-annotation pack, and the v0.6-dry reader check. Cost fields are removed and results reproduce exactly. The private motivating conversation is not included. Paper and data: CC BY 4.0; code: MIT licence.
Authors
- Vijay Suresh (ORCID: https://orcid.org/0009-0004-1471-0561)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-27
- DOI
- https://doi.org/10.5281/zenodo.23721945
- Primary Topic
- AI in Service Interactions
- Type
- preprint