Predicting Unfamiliar Solvers with Cross-Fitted Probability Pooling

How much can a few answers reveal about an unfamiliar AI solver? We start with a model of historical evaluation results, then update its predictions using answers from the solver and its peers. Our method pools question-level evidence, corrects for solver differences, and combines two updates according to held-out probability error. In a matched local benchmark-cold evaluation with ten solvers supplying evidence, it reaches Brier ALC 0.151. A separate oracle experiment compresses questions' historical solver outcomes into four informative coordinates. Recovering those coordinates for new questions remains an open problem.