Evaluating Pretrained Causal Language Models for Synonymy

The scaling of causal language models in size and training data enabled them to tackle increasingly complex tasks.Despite the development of sophisticated tests to reveal their new capabilities, the underlying basis of these complex skills remains unclear.We argue that complex skills might be explained using simpler ones, represented by linguistic concepts.As an initial step in exploring this hypothesis, we focus on the lexical-semantic concept of synonymy, laying the groundwork for research into its relationship with more complex skills.We develop a comprehensive test suite to assess various aspects of synonymy under different conditions, and evaluate causal open-source models ranging up to 10 billion parameters.We find that these models effectively recognize synonymy but struggle to generate synonyms when prompted with relevant context.