Opening Rankings from Engine Self-Play Are Less Certain Than They Look

Engine self-play allows chess openings to be compared under controlled conditions, without the selection effects of human game databases. When openings differ only slightly, however, rankings built from a few hundred games can be unstable. In Study 1, strength-limited Stockfish 15.1 played 17,400 games: ten forced White first moves and ten forced Black replies to 1.e4, each at three strength settings. The leading opening changed between settings, twice by a single game. There was no statistically reliable evidence, though, that differences among openings depended on strength (White p = .311, Black p = .843). The design could detect only single-opening effects of about 12 to 16 percentage points, and a single-opening effect of 12 points was detected in about 75% of simulated studies without correction (about 44% with a correction for ten openings). A Connect Four positive control detected an observed 22.5-point centercolumn advantage, demonstrating sensitivity to large opening differences. With 290 games per opening, the observed leader was the true best of ten openings in only about 44% of simulated studies assuming a between-opening standard deviation of three percentage points. Follow-up studies with complete game records showed that the settings were ordered in direct matches but did not form a consistent Elo scale, that win and draw rates differed across hardware and implementation configurations, and that an apparent 1.d3 weakness was not confirmed in new games. With Stockfish 19 and a fixed node budget, surveyselected replies showed no statistically reliable advantage over free engine choice. Opening leaderboards from a few hundred engine games can overstate certainty.