Traceable LLM-based validation of statements in knowledge graphs

• There is a paucity of research aimed at quality control of knowledge graphs, with LLMs providing a major opportunity. • We introduce a new LLM-based method for verifying KG statements, which is built on the paradigm that no internal knowledge of the LLM should be used in the verification process , but instead, the LLM is used solely as a tool to interpret and compare the verified statement with a supplied candidate grounding text. • Empirical evaluation is performed on two datasets and several LLM models. The first evaluation was performed using the BioRED dataset containing over 3000 statements. Using a Llama 3 model, we obtained an average precision of 88 % and an average recall of 44 % (LLama 70B model). The second evaluation was performed on 1000 randomly chosen records from the SNLI dataset, where the best accuracy of 89.5 % was obtained for Llama 3.1 405B model. • Two use cases are included that demonstrate how statements with data quality issues can be automatically identified in Wikidata and how the proposed approach can be used for verification, including two methodological options for the retrieval of grounding documents. • We provide source code and results for replication. This article presents a method for verifying RDF triples using LLMs, with an emphasis on providing traceable arguments. Because the LLMs cannot currently reliably identify the origin of the information used to construct the response to the user prompt, our approach is to avoid using internal LLM factual knowledge altogether. Instead, verified RDF statements are compared to chunks of external documents retrieved through a web search or Wikipedia. To assess the possible application of this retrieval augmented generation (RAG) workflow on biosciences content, we evaluated 1,719 positive statements from the BioRED dataset and the same number of newly generated negative statements. The resulting precision is 88 %, and recall is 44 %. This indicates that the method requires human oversight. We also evaluated the method on the SNLI dataset which allowed us to compare our approach with models specifically tuned for the natural language inference task. We demonstrate the method on Wikidata, where a SPARQL query is used to automatically retrieve statements needing verification. Overall, the results suggest that LLMs could be used for large-scale verification of statements in KGs, a task previously unfeasible due to human annotation costs.