Encoder Language Models for Zero-Shot Recommender Systems: Cross-Domain and Cross-Lingual Evaluation

This research provides deep insights into the capabilities of encoder models, offering a unique, controlled benchmark of over 20 state-of-the-art models. The aim is to analyze the viability of a potentially advantageous recommender architecture that capitalizes on the progress made in natural language processing (NLP) without the drawbacks of more advanced deep learning models or generative decoder-only large language models. We present a zero-shot recommender architecture used in a training-free setting. In this setting, item embeddings are aggregated into weighted user profiles and scored by cosine similarity against the full candidate pool. This approach makes use of state-of-the-art encoder language models in a frozen, inference-only, context-free state. We employed a temporal evaluation designed to simulate the cold-start problem, ordering user interactions over time and progressively revealing later interactions. We also used three contrasting datasets with specific challenges, as well as baseline methods of similar complexity and resource demands, such as collaborative filtering or content-based TF-IDF methods. For our recommendation task, we also compared the local large language model with the encoder models in a limited experiment. Our results suggest that frozen embedding-based recommendation could be a viable option, particularly when considering the cost/performance ratio. However, model and method selection should be evaluated against domain-specific tasks and protocols, rather than being based on general intuitions about scaling, general-purpose benchmark scores, or even leave-one-out validation methods.