Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability

Abstract LLMs have demonstrated strong language-learning and human-like response-generation capabilities, and they are increasingly used to support decision-making in high-risk sectors. However, their internal decision processes remain difficult to interpret, and their responses may lack transparency. The literature has explored numerous approaches to address transparency challenges in LLMs, including Neurosymbolic AI (NeSy AI). NeSy AI approaches were primarily developed for conventional neural networks and may not transfer directly to the distinctive characteristics of LLMs. Consequently, there is a limited systematic understanding of how symbolic AI can be effectively integrated into LLMs. This paper aims to address this gap by first reviewing established NeSy AI methods and then proposing a novel taxonomy of symbolic integration in LLMs, along with a roadmap to merge symbolic techniques with LLMs. The taxonomy organises the literature across four dimensions: (1) the stage of LLM development at which symbolic information is integrated; (2) the coupling mechanism; (3) the architectural paradigm; and (4) the algorithm-level or application-level perspective. The review identifies commonly used benchmarks, recent advances and important research gaps, and uses these findings to outline directions for future research. By highlighting the latest developments and notable gaps in the literature, it offers practical insights for implementing frameworks for symbolic integration into LLMs to enhance transparency.