Analyzing the Linguistic Priors of Language Models with Synthetic Languages

While modern language model architectures are often assumed to be language-agnostic, there is limited evidence as to whether these models actually process the vast diversity of natural languages equally well.We investigate this question by analyzing how well LMs learn carefully constructed artificial languages containing a variety of verbal complexity, ranging from simple paradigms to covering far more verb classes than occur in natural languages.Rather than learning all languages equally efficiently, models trained on these languages show strict preferences for processing simpler languages.Furthermore, while some observed model preferences mimic human linguistic priors, we find that they correspond to model memorization of its training data rather than generalization from it.This finding suggests that while model behavior often mimics human language understanding, the underlying causes of their proficiencies are likely very different.