Faster Is Not Stronger: A Case Study of Search Semantics, Simulator Fidelity and Throughput in a Fixed-Team gen9ubers Pokémon Agent
A faster simulator lets a game-playing agent search more positions within a turn. Does that extra search improve its choices? We studied a fixed-team agent for the Pokémon Showdown gen9ubers ladder, where each player chooses a move without seeing the opponent's choice or full team configuration. A Rust battle engine ran about 220 times faster per simulated turn than the reference simulator. After three rounds of fixes on the same 3,000 ladder positions, it matched the reference on 99.82% of deterministic one-turn outcomes. Agreement on a separate 80-position check before the final fixes was 98.1%. In a benchmark with 384 games per arm, the engine's native best-response search won 6–7 percentage points fewer games than the reference search despite completing roughly 1,400 times as many iterations. The two algorithms count different work as an iteration. A second benchmark ported the reference search, including its learned model of human move choice, to the faster engine. At the original node cap, the estimated win-rate difference was +1.4 percentage points (95% CI −2.1 to +4.9). With the full time budget it was +3.8 points (95% CI +0.5 to +7.1 before adjustment and −0.2 to +7.9 after adjustment for three comparisons). This benchmark completed 77% of its planned games. Across 2,463 fully logged ladder games, the introduction of human move models coincided with a higher estimated rating. Four policy updates with documented offline screening gains showed no detectable ladder gain. The benchmarks changed several features together, including how often the search could override the policy. They leave open which change accounts for the difference in playing strength.
Authors
- S M Abdulla Al Mamun
Institutions
- University of Manchester (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24374068
- Primary Topic
- Artificial Intelligence in Games
- Type
- preprint