Measuring the diversity of data and metadata in digital libraries

Abstract Diversity indices have been traditionally used to capture the biodiversity of ecosystems by measuring the effective number of species or groups of species. In contrast to abundance, which grows with the amount of data available and is sensitive to the appearance of small groups, diversity indices provide a more robust indicator on the variability of individuals. These types of indices can be employed in the context of digital libraries to analyse their content and metadata. They can be used, for example, to identify trends in the distribution of topics, to compare the lexica employed by different authors or to analyse the coverage of semantic metadata. In this article, the lexical diversity is measured through one of the most common indices employed to evaluate diversity, the Shannon index. The experiments show that this index slowly grows with the length of the text used to calculate it. As this growth has the diversity value as ceiling, the curves show that the true value of diversity will only be reached for very large samples. Unfortunately, the available text is often not long enough to achieve the convergence. This paper introduces therefore a new model for the calculation of the asymptotic value of the Shannon diversity of the vocabulary which outperforms traditional models. As regards metadata in digital libraries, we use the new model to analyse the topical specialization of a digital library and its time evolution and propose a more robust way to measure the variety of tags (classes and properties) employed by digital libraries to describe their holdings in Linked Open Data repositories.