Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing

Query Performance Prediction (QPP) estimates the effectiveness of retrieval systems for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, a critical gap remains in understanding how well QPPs generalize across diverse retrieval paradigms and collections, a question of robustness that has significant implications for their practical utility. This paper provides the first comprehensive cross-paradigm evaluation of QPP robustness and generalization capabilities, examining state-of-the-art QPPs including NQC, WIG, LETOR-based features, and newly explored dense-based predictors MQPPF and BERT-QPP. We systematically assess their performance across diverse sparse (BM25, DFree with and without query expansion), hybrid (SPLADE), and dense (ColBERT, TCT-ColBERT) rankers on four benchmark collections: TREC Robust, GOV2, WT10G, and MS-MARCO. The results reveal fundamental robustness challenges: predictors exhibit significant variability in accuracy, with collection being the dominant factor, followed by ranker type. Some sparse predictors perform adequately on specific collections such as TREC Robust and GOV2, but critically fail to generalize to other collections like WT10G and MS-MARCO. Dense-based predictors, while showing promise in specific scenarios with dense rankers, similarly lack generalization to sparse contexts. We demonstrate that these generalization failures severely limit practical applications: QPP-driven selective query processing achieves only marginal gains ( \(\approx\) 4% NDCG improvement), with reliability varying dramatically across settings. Our findings underscore that current QPP methods lack the robustness necessary for real-world deployment and highlight the urgent need for predictors that generalize reliably across diverse collections, align with modern dense retrieval architectures, and provide consistent utility for downstream applications. We publicly release our data and code to facilitate future research on robust QPP methods.