Performance and sustainability of BERT derivatives in dyadic data

In recent years, the Natural Language Processing (NLP) field has experienced a revolution, where numerous models –based on the Transformer architecture– have emerged to process the ever-growing volume of online text-generated data. This architecture has been the basis for the rise of Large Language Models (LLMs). Enabling their application to many diverse tasks in which they excel with just a fine-tuning process that comes right after a vast pre-training phase. However, their sustainability can often be overlooked, especially regarding computational and environmental costs. Our research aims to compare various BERT derivatives in the context of a dyadic data task while also drawing attention to the growing need for sustainable AI solutions. To this end, we utilize a selection of transformer models in an explainable recommendation setting, modeled as a multi-label classification task originating from a social network context, where users, restaurants, and reviews interact. • A detailed review of the pre-training process for each of the selected BERT variants. • A comparison of diverse transformers within an explainable recommendation setting, where dyadic data is modeled under a multilabel classification task using reviews as explanations. • A comparative result assessment considering environmental impact and performance of the models.