Beyond Precision and Recall: Measuring Search Engine Consistency Using Rank Stability

Traditional information retrieval metrics such as precision, recall, and F-measure assess document relevance but fail to capture the stability of search rankings. Search engines frequently update their algorithms, leading to variations in retrieved documents and their rankings across identical queries. This study introduces a novel rank correlation measure that quantifies search result consistency by analyzing rank fluctuations in repeated queries, serving as a proxy for temporally separated searches. The proposed approach distinguishes between overlapping and non-overlapping documents, providing a detailed measure of ranking stability. Empirical evaluations comparing Google and Bing reveal that Google exhibits greater rank fluctuations for non-overlapping documents, whereas both search engines maintain relatively stable rankings for overlapping documents. These findings highlight the importance of rank-based evaluation frameworks in search engine analysis, extending beyond conventional retrieval metrics. By incorporating stability assessments, this research enhances the methodology for evaluating search consistency in dynamic web environments, contributing to process innovation in information retrieval (SDG 9: Industry, Innovation, and Infrastructure) and promoting fair, reliable access to information (SDG 16: Peace, Justice, and Strong Institutions).