AttentionXML VS LLMs: An Empirical Evaluation of Extreme Multi-Label Classification Techniques

Extreme Multi-Label Classification (XMLC) plays a pivotal role in organizing and retrieving information in large-scale textual collections, by considering a very high number of potential labels for the documents. In this paper, we conduct an empirical evaluation of several XMLC approaches encompassing both dedicated techniques (AttentionXML and XR Transformer) and the use of Large Language Models (LLaMA2 7b Chat, LLaMA3 8b Instruct, and two Mistral models). We introduce both a new dataset based on OpenAlex as well as several new metrics to conduct our evaluations. Our results suggest that fine-tuning the LLMs using Low-Rank Adaptation significantly improves the performance of the models, bringing their results close to the ones of dedicated techniques. In the end, none of the method emerges as a clear winner, as picking the optimal XMLC technique heavily depends on the requirements of the use-case at hand.