A study on question answering in the ceramics domain using large language models with retrieved triples and generated textual knowledge prompts

The propensity of Large Language Models (LLMs) to hallucinate poses a significant challenge in specialized domains with fragmented knowledge, such as ceramic question answering. A critical yet underexplored factor is how the format of retrieved knowledge-structured versus unstructured-affects model accuracy. In this study, we develop a Retrieval-Augmented Generation (RAG) framework for the Chinese ceramic domain. A domain-specific knowledge graph was constructed from heterogeneous sources, and a BERT-based module was employed for hop prediction and path reasoning. The retrieved subgraphs were provided to LLMs either as structured triples or converted textual descriptions. We evaluate four LLMs (GPT-3.5, GPT-4, ChatGLM2-6B, LLaMA-2-7B) on a Chinese ceramic QA benchmark. Across all models, structured triples yield higher accuracy, with ChatGLM2-6B achieving the best performance (97.24%). Smaller models benefit more from structured representations, while larger models process textual descriptions more effectively. These results highlight the importance of knowledge representation for reducing hallucination in domain-specific QA. However, as all experiments are conducted on Chinese datasets within a single domain, the generalizability of our findings across languages and other vertical domains remains an open question. We discuss this limitation and outline directions for future multilingual and cross-domain validation.