Methods for the Automation of Subject Indexing from Simple Statistical to Large Language Models – a Case Study

Subject indexing, i.e., the enrichment of metadata records for textual resources with descriptors from a controlled vocabulary, is one of the core activities of libraries. Due to the proliferation of digital documents, it is no longer possible to annotate every single document intellectually, which is why libraries need to explore the potentials of automation on every level, up to and including methods from artificial intelligence (AI). While the current hype around deep learning approaches such as large language models (LLMs) has brought an unprecedented level of attention from the general public to AI, models from other branches of AI that were open source and showed potential for practical applications already started emerging over a decade ago. In this case study, we are investigating open source machine learning methods that are currently being used or evaluated in libraries for the automation of subject indexing, from statistical and lexical models to approaches that are based on transformer architectures, i.e., small and large language models. We present a selection of experimental results with regard to how well these models perform on typical bibliographic metadata of a library and draw conclusions on some of the difficulties surrounding the sizes and characteristics of bibliographic datasets as training data and what would have to be done in order to remedy the situation. We also describe possible measures for monitoring the quality of the output of such models, including automated solutions and approaches that integrate humans into the loop of quality control at various points. One promising approach is active learning, in which a coordinating algorithm decides which kind of input data would be most useful for training when annotated intellectually. Methods like these might gradually change the role of scientific librarians, which is why it is paramount to work together with subject indexing experts not only to ensure good quality output, but also to foster transparency and acceptance.