A Weighted Retrieval Model for Large Collections under the Exceedance over Relevant Extremes

Existing probabilistic retrieval models rank documents based on the normalized frequency of query terms.These models overestimate the relevance score of documents containing overrepresented terms that appear with extremely high frequency in large web document collections.The past work on Information Retrieval (IR) targeting this problem uses truncated distributions to mitigate the impact of high-frequency terms in the ranking function.However, its effectiveness heavily relies on the selection of the correct truncation point, as an incorrect choice can lead to aggressive truncation and result in missing many relevant documents.In this work, we propose a weighted probabilistic retrieval model by modeling the probability of exceedance over the maximum values of normalized term frequencies in relevant documents of a set of training queries.The proposed model assigns weights in the ranking function, giving lower weights to overrepresented terms and higher weights to other terms.We conduct experiments on large collections and show that the proposed model outperforms state-of-the-art retrieval models, especially in datasets with high-frequency terms and spam.The proposed model is capable of reducing the effect of overrepresented terms in the ranking function using a soft scoring method, thereby preventing the rejection of relevant documents from the ranked list.