How effective are web search engines? A metric-based comparative study of Google, Bing and Yandex for informational queries in business studies

Purpose This study aims to comparatively evaluate the retrieval performance of select web search engines, i.e. Google, Bing and Yandex, based on standardised metrics. Design/methodology/approach The study draws informational queries from the discipline of business studies using titles, keywords and abstracts of PhD theses from the Shodhganga repository. It categorises queries into one-word, two-word and complex-word queries and evaluates them using pre-established relevance criteria on a three-point relevance scale to calculate precision, mean average precision, relative recall, F-measure and fall-out ratios. The study further employs ANOVA to determine the statistical significance of the difference in means between the above-stated metrics for Google, Bing and Yandex, to understand the performance of one web search engine compared to the others. Findings The analysis of variance (ANOVA) results indicate that the difference between the means is not statistically significant for Google, Bing and Yandex in terms of precision, relative recall, F-measure and fall-out ratio. However, descriptive statistics, although not statistically significant, revealed a superiority of Google over the other two search engines in terms of mean average precision, mean relative recall, F-measure and fall-out ratio. Additionally, it was found that complex-word queries performed better than one-word and two-word queries for all three web search engines. Research limitations/implications The paper studies the retrieval effectiveness of only three web search engines, among a pool of search engines. It restricts its use to only informational queries, leaving aside the navigational and transactional queries. Future studies can target a broader range of web search engines, utilising all three categories of queries, including navigational and transactional queries, across multiple disciplines. The study has implications for users, web search engine programmers and Librarians. It will help users, especially academic users in the field of business studies, to identify more suitable web search based on the results of the metrics, as well as help programmers to understand which retrieval techniques work better and thereby improve the efficiency of search engines better. Moreover, librarians include topics such as retrieval effectiveness in web search, query formulation strategies, keyword mappings, etc., in their library orientation programs. Originality/value The study provides an original empirical evaluation of the performance of web search engines Google, Bing and Yandex by conducting a statistical comparison of their performance on informational queries drawn from the field of business studies. It also compares the select web search engines based on query types.