Knowledge Integrity in Large Language Models: A State-of-The-Art Review

Large Language Models (LLMs) are emerging technologies and a growing research trend in Artificial General Intelligence (AGI), which envisions a future where machines can think and learn like humans across a wide range of tasks. Information generated by LLMs is essentially the prediction of next tokens in Natural Language Processing (NLP) tasks. However, the generated content is always subject to issues of truthfulness and hallucinations. The information and knowledge integrity of LLM-generated content therefore remains subjective. Exploring recent literature on the integrity of LLMs in a systematic manner is both timely and essential. Moreover, ensuring the reliability of LLMs in real-world applications is critical. Various approaches have been explored to promote information and knowledge integrity in LLMs, including adversarial training, data augmentation, and calibration methods. However, beyond these techniques, other strategies also contribute to maintaining knowledge integrity. This paper specifically focuses on three such approaches: knowledge distillation, semantic integrity, and provenance tracking, which play essential roles in ensuring that LLMs generate accurate, consistent, and trustworthy information. Knowledge distillation enhances model efficiency by transferring knowledge from larger models to smaller ones while preserving essential learning without compromising knowledge integrity. This reduces hallucinations. Semantic integrity safeguards consistency and strengthens the robustness of generated outputs. It is concurrently checking the meaningfulness of the outputs with the context. Provenance tracking improves transparency and trustworthiness through mechanisms such as data lineage and explainability, thereby ensuring the credibility of the LLM-generated responses. This review suggests that knowledge distillation, semantic integrity, and provenance tracking can enhance the reliability of LLM outputs, with prior studies reporting reductions in hallucination rates, improvements in robustness, and gains in factual consistency.