Evaluating Large Language Models Via Multi-Modal User Knowledge Graphs: A Comprehensive Assessment Framework

Large language models (LLMs) have been widely adopted across various industries, but issues such as hallucinations, biases, and erroneous outputs frequently arise, compromising their reliability and safety. Objectively assessing how well an LLM interprets user queries is crucial for selecting the right model for practical problem-solving. This paper proposes an evaluation method that leverages user knowledge graphs to measure an LLM's comprehension of user input. Specifically, we construct both text-based and graphical test cases derived from a user knowledge graph, thereby enabling multi-modal assessment of the LLM's understanding. To implement our approach, a knowledge graph is first built from the user's input and then mutated to produce diverse test cases. A search-based testing method is then applied to evaluate the model's comprehension. We provide a case study demonstrating the framework. Our findings indicate that multi-modal test cases outperform purely text-based test cases in revealing the true understanding capability of LLMs. Among the eight models tested, DeepSeek and Doubao exhibit stronger comprehension than the remaining six.