Can Large Language Models Understand Chinese Neologisms?

In recent years, large language models (LLMs) have achieved groundbreaking progress in natural language processing, demonstrating remarkable generalization capabilities in text comprehension, content generation, and semantic reasoning. However, language itself constantly evolves with frequent emergence of neologisms, making models' ability to accurately understand their meanings and make contextually appropriate inferences a crucial indicator of their generalization capacity. To investigate LLMs' comprehension of Chinese neologisms, this study constructs an evaluation framework specifically designed to assess models' understanding and application capabilities regarding Chinese neologisms. We curated a dataset containing 2,644 evaluation items from 645 neologisms selected from the Annual Media Neologism Lists (2021–2023) in China's Language Life Reports. The evaluation results reveal substantial room for improvement in large language models' comprehension and application of Chinese neologisms compared to their performance with common words. The DeepSeek model, predominantly trained on Chinese corpora, demonstrates significant advantages in processing Chinese neologisms, while ChatGPT -40 exhibits stronger adaptive learning capabilities. Although DeepSeek and ChatGPT -40 show varying performance across different neologism categories, their accuracy distributions follow similar patterns. Notably, both models' accuracy in understanding neologisms does not show a year-by-year decline but rather displays a performance trough in 2022. This phenomenon stems from the uneven distribution of neologism types across years - the higher proportion of abbreviated neologisms in 2022 (compared to other years) particularly constrained models' comprehension. The analysis indicates that semantic neologisms and abbreviated neologisms represent the most challenging categories, suggesting critical improvement opportunities for large language models.