Large Language Models as Judges for Human Subject Indexing

The study evaluates LLMs as judges for human subject indexing by comparing their performance against a manually curated gold standard across descriptions of forty Polish library and information science papers. The metrics—accuracy, Cohen’s kappa, precision, recall, and F1 score—yielded moderate (0.56) to high values (0.96). Moreover, the study found that AI–gold-standard disagreements that coincided with disagreements between the human evaluators were more frequent for Gemini (66.67%) than for ChatGPT (39.13%). The LLMs suffer from deficiencies evaluating incorrectly assigned descriptors, which hinders their adoption in real-world library settings.