Probing Fine-Grained Hierarchical Concept Comprehension and Generation in Large Language Models

Conceptual knowledge is fundamental to human cognition, and the unprecedented performance of large language models (LLMs) necessitates being tested for conceptual knowledge. However, many existing efforts focus on the overall assessment of conceptual knowledge, often overlooking fine-grained concept relationships, particularly the hypernym-hyponym relationship. In this work, we probe LLMs' concept understand and generations by various granularities hypernyms. Specifically, we design two probing tasks, hypernym judgment and hypernym generation, to assess the comprehension and generation of hypernym relationships in LLMs. These tasks are evaluated using a novel metric, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">relative hypernym granularity</i>, that quantifies the conceptual semantic distance to the query concepts. Additionally, We probe the semantic-enhanced prompts to enhance the reasoning abilities of LLMs, which is conducted by two structural hypernyms. Our experiments reveal that the latest language models, such as ChatGPT and Llama2, demonstrate consistent competence in managing fine-grained hypernyms. Notably, this research provides an in-depth analysis of hypernym hallucination, offering new perspectives and contributing meaningfully to the understanding of conceptual knowledge in LLMs. Our work delivers a comprehensive evaluation of concept granularity, that simultaneously covers (i) fine-grained hypernym judgement & generation, (ii) structured semantic prompting, and (iii) hypernym hallucination, providing the systematic evaluation of concept understanding capacity in LLMs.