Accuracy Is Not Strength: Learned Models Under Fixed Time Budgets in Imperfect-Information Search
A learned component can predict human moves or game outcomes more accurately without making a search agent stronger. We examine this gap through recorded model interventions in Pokémon singles and controlled tests of model width and placement. In the ledger's same-role replacement group, none of 16 changes produced a detectable gain; two lost 9.0 and 24.9 percentage points despite better offline metrics. Some changes to acting-policy training did improve win rate. A pre-specified width experiment raised held-out move accuracy from 57.0% to 58.0%; the widest model ran about a sixth as many search iterations as the 64-wide model at every budget, with no detectable width effect on win rate after accounting for search size. The largest loss replicated at −28.0 points and remained −23.5 points with similar measured search size, weakening a latency explanation. A replay audit showed that much of the target leaf's offline advantage came from hindsight information. In a placement experiment, the same behaviour model added 9.8 points [+5.4, +14.1] as a root prior at unchanged search throughput and 3.8 [−0.6, +8.1] inside the tree. A simple cost accounting predicts how placement changes the price of inference; the measurements indicate that this price alone does not account for strength. The paper is a case study of existing search and learning methods, not a new method.
Authors
- S. M. Abdulla Al Mamun
Institutions
- University of Manchester (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24373854
- Primary Topic
- Artificial Intelligence in Games
- Type
- preprint