Effects of Weighting Schemes on Retrieval Precision

Finding relevant information in a vast and growing amount of data has become significant since the arrival of the internet. An Information Retrieval System is described as searching and retrieving a list of documents, such as web pages or other items, in response to a user query. There are many weighting schemes, such as Boolean, Term Frequency or TF, and Term-Frequency-Inverse-Document-Frequency or TFIDF. In Boolean, a ‘1’ indicates the presence of a term in the document, whereas a ‘0’ indicates its absence. In TF, the term is represented by the number of occurrences in a document. TFIDF is the most popular one. It combines TF with an inverse document frequency IDF; IDF gives less weight to terms that frequently occur in many documents. With reference to the information retrieval system literature, TFIDF gives more accurate results on the computationally expensive expenses. Term Frequency is less expensive than TFIDF, and the least expensive one is the Boolean weighting scheme. In this paper, we investigated the effect of different weighting schemes and found out that, in many cases, Boolean and TF performed the same as TFIDF and, in a few cases, outperformed them. The experiments in this paper were conducted on many datasets of variant sizes and types. The dataset documents were indexed using TFIDF, TF, and Boolean. Then, using the queries that come with the dataset, we computed the precision and recall by comparing the results of different weighting schemes; the queries were also indexed using TFIDF, TF, and Boolean weighting schemes. The objective is to show that the use of the Boolean or TF weighting scheme, which is considered not computationally expensive, instead of TFIDF, which is regarded as very computationally expensive, does not significantly affect the results and, in a few cases, gives better results.