LIBERTUS: A Pretrained Multilingual Model for Cross-Lingual NLP Tasks

This paper presents Team Allen AI's system, LIB-ERTUS, a pretrained multilingual model fine-tuned for various downstream NLP tasks across multiple languages. The model demonstrates decent performance, achieving F1-scores of 70% or higher for most language-task pairs, highlighting its effectiveness in multilingual contexts. Despite this, the system struggles with subtokens of multiword expressions due to its training paradigm. Notably, I explored the cross-lingual capabilities of LIBERTUS, particularly in lemmatization, showing that transfer learning is viable with limited corpora. The limitations of my approach include the size of the pretrained model, the lack of a balanced token distribution strategy during pretraining, and the treatment of label combinations as individual classes, which restricted the model's ability to predict unseen combinations. Future work includes addressing these limitations to enhance performance.