Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation
Background We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged. Objective We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates. Methods Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations. Results Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis. Conclusions Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.
Authors
- Taichi IKEZAKI (ORCID: https://orcid.org/0009-0003-4902-9218)
- Joe Hasei (ORCID: https://orcid.org/0000-0002-4028-0786)
- Yuichiro Toda (ORCID: https://orcid.org/0000-0003-4170-2300)
- Ryuichi Nakahara (ORCID: https://orcid.org/0000-0001-8333-7378)
- Koichi Takeuchi
- Yuki Nishimura
- Hiroki Matsuoka (ORCID: https://orcid.org/0009-0008-9676-0739)
- Toshifumi Ozaki
Publication Details
- Journal
- Frontiers in Digital Health
- Published
- 2026-11-18
- DOI
- https://doi.org/10.3389/fdgth.2026.1981579
- Primary Topic
- Information Retrieval and Search Behavior
- Type
- article
- Field-Weighted Citation Impact
- 0.00