Is more always better? Measuring the quality of ranking data through information entropy

ABSTRACT Purpose Rank aggregation plays a crucial role in various academic and practical applications. However, accurately assessing the quality of ranking data remains a critical challenge. This study aims to propose methods for assessing the quality of ranking data from the perspective of its distribution. Design/methodology/approach This study adopts a network science perspective, transforming ranking data into a network and evaluating its quality using network structural entropy. In addition, we extended three commonly used ranking data generation models to produce ranking data with different distribution characteristics. Finally, the effectiveness of the proposed methods was validated using both synthetic and real-world data. Findings Through experiments, we validated the effectiveness of the proposed methods in assessing the quality of ranking data from the perspective of distribution. Additionally, the study revealed the following: (1) simply increasing the number of input rankings does not necessarily improve data quality; (2) when dealing with unevenly distributed ranking data, different aggregation methods exhibit significant differences in performance; and (3) increasing the length of input rankings can mitigate the decline in aggregation effectiveness caused by the uneven probability of each object being ranked. Research limitations (1) This study focuses on the impact of distribution characteristics on the quality of ranking data, without considering the effect of disagreements within the data; (2) although the proposed methods have been validated on synthetic and real-world datasets, their generalizability may still require further testing on more diverse datasets. Practical implications The methods proposed in this study enables researchers and information managers to more accurately assess the quality of input data before performing rank aggregation, thereby enhancing decision-making reliability. Originality/value This study proposes two novel methods from the perspective of network science to address the challenge of data quality assessment in rank aggregation, providing both theoretical support and practical insights for related fields.