Never Miss the Narrows: What a Short Instruction Does and Doesn't Fix When AI Assistants Plan Trips Over Long Conversations

AI assistants now help people plan multi-day trips over long conversations. They can say something true and important once, early on, and then fail to raise it again when the traveler actually decides. They can also quietly trade away a must-see to fit something else, or forget a refund deadline they noted themselves. We call the goal of avoiding these failures informed-choice preservation. We propose a model-agnostic instruction block, the "Never Miss the Narrows" protocol: six scored rules plus one hygiene rule. We test it against a plain baseline and a variant with explicit memory. What we tested: 7 models (4 frontier, 3 open-weight) on 36 scenarios, preregistered and independently reviewed before data collection: long conversations of 120–248 messages at real places, with scripted (frozen) middles and live replies at six decision points each; short cases derived from documented real incidents; constructed cases; and no-nag controls. What the protocol fixes. The most meaningful measure is what a traveler understands after reading a reply, so we asked AI "travelers" to read each reply cold and say what they could now choose and what each choice would gain or cost. Points understood rose from 0.42 to 0.59 (+0.17, 95% CI 0.12–0.23), in all six long cases. Two gains came through clearly. Deadlines the assistant had recorded itself: 58% → 71% of readers knew the deadline and why the booking might not fit; judges and readers agreed almost perfectly on this. Spare time: readers who realised the day would end early went from 6% to 38%. The preregistered judge composite shows a larger gain, 0.49 → 0.76 (+0.27, 95% CI 0.19–0.35). Part of that gap is rubric echo: some judge metrics mirror the protocol's own wording. What it doesn't fix. Models mentioned the must-see far more often, but only 30% of readers could say why it mattered even after protocol replies. Mentioning is not informing. The explicit-memory design we tested, a model-written JSON summary, added little (+0.03 to +0.04) while using about three times the compute. Possible costs. In controls where nothing was critical, the judges rated substantial over-intervention up from 10% to 26% (35% with memory). However, AI readers barely more often wanted to reopen a settled decision (+0.03 to +0.04, not significant), so the size of this cost is uncertain. Decision-critical facts the protocol does not mention were disclosed slightly less often. Replies grew 11–14% longer. Replay of a real failure. Re-running the conversation that motivated this work, current frontier models no longer repeated the original mistake (cutting a walk short before its signature endpoint). The gap that remains is narrower: why the endpoint matters, the self-recorded cancellation deadline, and idle time. Without the protocol, no model raised the deadline. Caveat. All labels come from AI models (three judges, two readers, one grader); human annotation is in progress. We compared the protocol only with a plain baseline, not with simpler disclosure prompts from prior work. The six long cases are scripted around the moments the protocol targets. Working paper v4.3. The attached package (ICP-NeverMissTheNarrows-v0.5.3-public.zip) contains the preregistered evaluation: 36 scenarios, 755 runs across 7 models, judge labels, analysis code, a blinded human-annotation pack, and the v0.6-dry reader check. Cost fields are removed and results reproduce exactly. The private motivating conversation is not included. Paper and data: CC BY 4.0; code: MIT licence.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-27
DOI
https://doi.org/10.5281/zenodo.23721945
Primary Topic
AI in Service Interactions
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Never Miss the Narrows: What a Short Instruction Does and Doesn't Fix When AI Assistants Plan Trips Over Long Conversations

Vijay Suresh
Zenodo (CERN European Organization for Nuclear Research)
AI in Service Interactions
preprint

Never Miss the Narrows: What a Short Instruction Does and Doesn't Fix When AI Assistants Plan Trips Over Long Conversations

Vijay Suresh
preprint en

Abstract

AI assistants now help people plan multi-day trips over long conversations. They can say something true and important once, early on, and then fail to raise it again when the traveler actually decides. They can also quietly trade away a must-see to fit something else, or forget a refund deadline they noted themselves. We call the goal of avoiding these failures informed-choice preservation. We propose a model-agnostic instruction block, the "Never Miss the Narrows" protocol: six scored rules plus one hygiene rule. We test it against a plain baseline and a variant with explicit memory. What we tested: 7 models (4 frontier, 3 open-weight) on 36 scenarios, preregistered and independently reviewed before data collection: long conversations of 120–248 messages at real places, with scripted (frozen) middles and live replies at six decision points each; short cases derived from documented real incidents; constructed cases; and no-nag controls. What the protocol fixes. The most meaningful measure is what a traveler understands after reading a reply, so we asked AI "travelers" to read each reply cold and say what they could now choose and what each choice would gain or cost. Points understood rose from 0.42 to 0.59 (+0.17, 95% CI 0.12–0.23), in all six long cases. Two gains came through clearly. Deadlines the assistant had recorded itself: 58% → 71% of readers knew the deadline and why the booking might not fit; judges and readers agreed almost perfectly on this. Spare time: readers who realised the day would end early went from 6% to 38%. The preregistered judge composite shows a larger gain, 0.49 → 0.76 (+0.27, 95% CI 0.19–0.35). Part of that gap is rubric echo: some judge metrics mirror the protocol's own wording. What it doesn't fix. Models mentioned the must-see far more often, but only 30% of readers could say why it mattered even after protocol replies. Mentioning is not informing. The explicit-memory design we tested, a model-written JSON summary, added little (+0.03 to +0.04) while using about three times the compute. Possible costs. In controls where nothing was critical, the judges rated substantial over-intervention up from 10% to 26% (35% with memory). However, AI readers barely more often wanted to reopen a settled decision (+0.03 to +0.04, not significant), so the size of this cost is uncertain. Decision-critical facts the protocol does not mention were disclosed slightly less often. Replies grew 11–14% longer. Replay of a real failure. Re-running the conversation that motivated this work, current frontier models no longer repeated the original mistake (cutting a walk short before its signature endpoint). The gap that remains is narrower: why the endpoint matters, the self-recorded cancellation deadline, and idle time. Without the protocol, no model raised the deadline. Caveat. All labels come from AI models (three judges, two readers, one grader); human annotation is in progress. We compared the protocol only with a plain baseline, not with simpler disclosure prompts from prior work. The six long cases are scripted around the moments the protocol targets. Working paper v4.3. The attached package (ICP-NeverMissTheNarrows-v0.5.3-public.zip) contains the preregistered evaluation: 36 scenarios, 755 runs across 7 models, judge labels, analysis code, a blinded human-annotation pack, and the v0.6-dry reader check. Cost fields are removed and results reproduce exactly. The private motivating conversation is not included. Paper and data: CC BY 4.0; code: MIT licence.

Zenodo (CERN European Organization for Nuclear Research)
AI in Service Interactions
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon