An Invasive Embedding Model in Favor of Low-Resource Languages Understanding

The contextual representations generated by multilingual pre-trained language models involve semantic content that conveys the meaning of sentences, as well as language-dependent nuances that indicate language-specific information. However, for cross-lingual machine learning tasks, particularly those suffering from data scarcity, the language-specific information blended in the contextual representation not only increases complexity but also compromises performance. This challenge is especially critical for natural language understanding (NLU) tasks such as intent detection (ID) and slot filling (SF), which are fundamental for the functionality of multilingual dialogue systems. Central to the idea is eradicating language-specific information from the contextual representation generated by an encoder through an adversarial manner, while preserving semantic information through input reconstruction via a decoder. In this regard, we propose an encoder-decoder model that employs adversarial learning techniques to enhance knowledge transferability across diverse languages for cross-lingual NLU tasks. Experimental results on two publicly available datasets, Facebook-multilingual (XTOD) and Persian-ATIS, demonstrate that our model significantly outperforms its main counterparts and achieves competitive results compared to state-of-the-art models across diverse languages in zero-shot scenarios. Notably, our findings highlight that achieving language-independent representations through adversarial learning followed by multi-task learning improves the model's performance in terms of accuracy and F1-score for NLU tasks.