Learning Context-aware Term Importance for Query Performance Prediction

Ad hoc retrieval, a cornerstone task in Information Retrieval (IR) , aims to rank documents in response to a user’s query, often without prior knowledge of the user’s specific information need. While transformer-based neural rankers have achieved state-of-the-art performance in ad hoc retrieval, their effectiveness varies significantly across queries. Certain queries—commonly referred to as hard queries —remain particularly challenging, highlighting critical gaps in retrieval models. Identifying these hard queries is essential for improving retrieval systems, motivating the task of Query Performance Prediction (QPP) , which aims to estimate the effectiveness of a query without requiring access to relevance judgments. In this article, we propose Context-aware Query Performance Prediction ( CA-QPP ) , a novel post-retrieval QPP method, which builds on the foundations of perturbation-based QPP methods that hypothesize a relationship between query sensitivity to small perturbations and query retrieval effectiveness. Building on this foundation, our approach exposes the given query to perturbations by constructing two query variations: an effective variation emphasizing terms that enhance retrieval and an ineffective variation accentuating terms that hinder it. By contrasting the retrieval outcomes of these variations using a cross-encoder model, CA-QPP captures the interplay of term contributions and predicts the performance for the given query. We evaluate CA-QPP on the widely used MS MARCO datasets and their associated query sets, including TREC DL 2019 , TREC DL 2020 , DL-Hard , TREC DL 2021 , and TREC DL 2022 , which feature extensive human-labeled relevance judgments. Our experiments demonstrate that CA-QPP consistently outperforms traditional and neural-based QPP baselines across standard correlation metrics, including Pearson’s \(\rho\) , Kendall’s \(\tau\) , and Spearman’s \(\rho\) . Through a detailed case study, we further illustrate the mechanics of CA-QPP and provide empirical evidence for its ability to model the contextual impact of individual query terms, making it a robust framework for query performance prediction.