The Role of Abstract Representations and Observed Preferences in the Ordering of Binomials in Large Language Models

To what extent do large language models learn abstract representations as opposed to more superficial aspects of their very large training corpora?We examine this question in the context of binomial ordering preferences involving two conjoined nouns in English.When choosing a binomial ordering (radio and television vs television and radio), humans rely on more than simply the observed frequency of each option.Humans also rely on abstract ordering preferences (e.g., preferences for short words before long words).We investigate whether large language models simply rely on the observed preference in their training data, or whether they are capable of learning the abstract ordering preferences (i.e., abstract representations) that humans rely on.Our results suggest that both smaller and larger models' ordering preferences are driven exclusively by their experience with that item in the training data.Our study provides further insights into differences between how large language models represent and use language and how humans do it, particularly with respect to the use of abstract representations versus observed preferences.