Accuracy Is Not Strength: Learned Models Under Fixed Time Budgets in Imperfect-Information Search

A learned component can predict human moves or game outcomes more accurately without making a search agent stronger. We examine this gap through recorded model interventions in Pokémon singles and controlled tests of model width and placement. In the ledger's same-role replacement group, none of 16 changes produced a detectable gain; two lost 9.0 and 24.9 percentage points despite better offline metrics. Some changes to acting-policy training did improve win rate. A pre-specified width experiment raised held-out move accuracy from 57.0% to 58.0%; the widest model ran about a sixth as many search iterations as the 64-wide model at every budget, with no detectable width effect on win rate after accounting for search size. The largest loss replicated at −28.0 points and remained −23.5 points with similar measured search size, weakening a latency explanation. A replay audit showed that much of the target leaf's offline advantage came from hindsight information. In a placement experiment, the same behaviour model added 9.8 points [+5.4, +14.1] as a root prior at unchanged search throughput and 3.8 [−0.6, +8.1] inside the tree. A simple cost accounting predicts how placement changes the price of inference; the measurements indicate that this price alone does not account for strength. The paper is a case study of existing search and learning methods, not a new method.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24373854
Primary Topic
Artificial Intelligence in Games
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Accuracy Is Not Strength: Learned Models Under Fixed Time Budgets in Imperfect-Information Search

S. M. Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Games
preprint

Accuracy Is Not Strength: Learned Models Under Fixed Time Budgets in Imperfect-Information Search

S. M. Abdulla Al Mamun
preprint en

Abstract

A learned component can predict human moves or game outcomes more accurately without making a search agent stronger. We examine this gap through recorded model interventions in Pokémon singles and controlled tests of model width and placement. In the ledger's same-role replacement group, none of 16 changes produced a detectable gain; two lost 9.0 and 24.9 percentage points despite better offline metrics. Some changes to acting-policy training did improve win rate. A pre-specified width experiment raised held-out move accuracy from 57.0% to 58.0%; the widest model ran about a sixth as many search iterations as the 64-wide model at every budget, with no detectable width effect on win rate after accounting for search size. The largest loss replicated at −28.0 points and remained −23.5 points with similar measured search size, weakening a latency explanation. A replay audit showed that much of the target leaf's offline advantage came from hindsight information. In a placement experiment, the same behaviour model added 9.8 points [+5.4, +14.1] as a root prior at unchanged search throughput and 3.8 [−0.6, +8.1] inside the tree. A simple cost accounting predicts how placement changes the price of inference; the measurements indicate that this price alone does not account for strength. The paper is a case study of existing search and learning methods, not a new method.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Artificial Intelligence in Games
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon