Predicting AI Performance on Unseen Benchmarks Using a Hierarchical Bayesian Model: PAIEC 2026 Technical Report

This technical report presents the research and development of a hierarchical Bayesian prediction system for the AIMS Predictive AI Evaluation Competition (PAIEC 2026), associated with NeurIPS 2026. The research addresses the problem of predicting the probability that an AI system correctly answers questions from previously unseen benchmarks under limited feedback. The evaluation setting involves making initial predictions without labels and subsequently updating predictions as selectively revealed labels become available. Methodology and Technical Contributions • Developed a compact hierarchical Bayesian prediction model incorporating benchmark-level offsets with heavy-tailed priors, pair-level ability, item difficulty pooled across subjects, and item-group effects. • Implemented exact posterior integration and Bayesian pooling to support prediction under sparse feedback and previously unseen benchmark conditions. • Designed a systematic experimental framework incorporating baseline comparisons, leave-one-benchmark-out (LOBO) evaluation, calibration analysis, leakage forensics, signal decomposition, and adversarial red-team experiments. • Established preregistered experiments, explicit statistical decision rules, reproducibility checks, automated validation, and competition submission pipelines. • Implemented the final prediction system using Python and NumPy. Selected Experimental Results • Local leave-one-benchmark-out evaluation (one subject per benchmark, 10 paired seeds): achieved a Brier ALC of 0.2088, compared with 0.2451 for the official baseline. • Official formative evaluation: achieved a mean Brier ALC of 0.1839 across four runs (standard deviation: 0.024). Lower Brier ALC indicates better performance. The local LOBO evaluation and official formative evaluation represent distinct experimental settings and should not be interpreted as a final competition ranking. Reproducibility and Research Artifacts The accompanying repository provides model implementations, experimental scripts, evaluation utilities, automated tests, competition submission artifacts, a technical report, and instructions for reproducing the reported results. The underlying gated measurement dataset is not redistributed and requires separate access. Repository: https://github.com/Shashwat-srivastav/s2wish-paiec2026 Keywords: Bayesian Statistics, Hierarchical Bayesian Modeling, Artificial Intelligence Evaluation, Benchmark Generalization, Probabilistic Prediction, Machine Learning, Statistical Calibration, Experimental Design, Reproducible Research.