Knowledge Breadth and Depth Measurement of Large Language Models ( <scp>LLMs</scp> )

ABSTRACT This study constructed an integrated framework for measuring LLMs' knowledge breadth (able to cover multiple fields) and depth (able to handle complex problems) and perform calculations based on evidence. The current research evidence on measuring LLMs was obtained through a systematic review. Then, an integrated framework based on the matching of capability and task was constructed, where the generation capability and knowledge management task reflect the knowledge breadth, while other capabilities and tasks reflect the knowledge depth. Knowledge breadth and depth were measured through Coverage and Revealed Technological Advantage. The results showed that 36 matches of capability and task were selected from representative papers, and the knowledge breadth and depth panorama of LLMs was displayed. Furthermore, there is no absolute advantage or disadvantage in LLMs. This study clarifies the boundaries of the LLM's measurement, and the framework further ensures the diversity of benchmark models and avoids redundancy and misalignment biases.