Chinese Large Language Models Evaluation in the Field of Scientific and Technical Information

ABSTRACT The field of scientific and technical information (STI) is characterized by limited open‐source training data, strong timeliness requirements, and high demands for professional expertise, necessitating the combination of general capabilities and domain‐specific capabilities of large language models (LLMs). Based on common STI research tasks, this study establishes benchmark datasets for evaluating LLMs in the STI field, comprising basic knowledge ability, dynamic research ability, and thematic research ability. A total of 1,557 objective and subjective questions were selected to evaluate the performance of eight LLMs developed by commercial organizations, research institutions, and universities in the STI field. The results indicate that LLMs perform well in terms of STI domain knowledge but still exhibit significant gaps in dynamic and thematic information research. There is a need to actively explore and promote the integration, adaptation, and application of LLM technologies in the STI field to provide robust support for high‐quality and high‐efficiency STI services.